Method, device, storage medium, and program product for checking and highlighting duplicate text

Abstract

The disclosed embodiments disclose a method, device, storage medium, and program product for checking for duplicate text and highlighting it. The method includes: receiving a target text, performing structural analysis on the target text, and separating the original format and the first plain text; obtaining a repeated segment between the first plain text and a preset text based on a first length threshold and a second length threshold; locating the start and end positions of the repeated segment in the first plain text; adding a highlight mark to the first plain text according to the start and end positions to form a text structure with a highlight mark; integrating the text structure with the original format to generate a formatted text; and displaying the formatted text. The method can improve the accuracy of checking for duplicate text while ensuring that the original format of the target text is not destroyed, thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language processing, and in particular to a method, device, storage medium, and program product for checking and highlighting text duplication. Background Art

[0002] With the widespread use of digital documents, the importance of text duplication detection in academic research, content creation, legal document review and other fields has become increasingly prominent. Traditional duplication detection tools can quickly check text for duplicate content and highlight it after detecting duplicate content, so that users can quickly locate and process it, but this highlighting often conflicts with the original format of the text. However, this highlighting function has significant limitations in practical applications. Since the design of traditional duplication detection tools focuses on rapid duplication detection and highlighting, it often ignores the issue of retaining the original format of the text, resulting in a conflict between the highlighted part and the original format of the text. This conflict may manifest as the highlighted part covering or affecting the original editing format of the text, such as font, style, layout, etc., thereby interfering with the user's document editing habits. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a method, device, storage medium, and program product for checking for duplicate text and highlighting it, which can improve the accuracy of checking for duplicate text while ensuring that the original format of the target text is not destroyed, thereby improving the user experience.

[0004] In a first aspect, the present disclosure provides a method for checking for duplicate text and highlighting the duplicate text, which adopts the following technical solution:

[0005] Receiving a target text, performing structural analysis on the target text, and separating an original format and a first plain text;

[0006] Based on a first length threshold and a second length threshold, obtaining a repeated segment between the first plain text and a preset text;

[0007] Locating the start and end positions of the repeated segment in the first plain text;

[0008] Adding highlight marks to the first plain text according to the start and end positions to form a text structure with highlight marks;

[0009] Integrate the text structure with the original format to generate formatted text;

[0010] The format text is displayed.

[0011] Optionally, the receiving the target text, performing structural analysis on the target text, and separating the original format and the first plain text includes:

[0012] receiving the target text through an interface service, and performing a security check on the target text;

[0013] If the security check passes, a preset separation tool is used to convert the text content in the target text into the first plain text, and the original format is extracted from the target text and stored in a database;

[0014] If the security check fails, a text optimization prompt is transmitted to the user terminal device.

[0015] Optionally, acquiring a repeated segment between the first plain text and a preset text based on a first length threshold and a second length threshold includes:

[0016] Performing coarse-grained semantic segmentation on the first plain text to obtain multiple semantic segments;

[0017] Performing fine-grained semantic division on the multiple semantic segments to obtain multiple sentences;

[0018] Based on a first length threshold and a second length threshold, length-standardizing the plurality of sentences to obtain a plurality of standard short sentences;

[0019] Extracting content from the preset text to obtain a second plain text;

[0020] dividing the second plain text into a plurality of segments;

[0021] Obtaining the similarity between the standard short sentence and each paragraph;

[0022] When the maximum similarity is not less than a preset similarity threshold, obtaining the number of consecutively repeated words of the standard short sentence;

[0023] When the number of consecutively repeated words is not less than a preset word count threshold, the standard short sentence is determined to be the repeated segment.

[0024] Optionally, the length of the plurality of sentences is normalized based on the first length threshold and the second length threshold to obtain a plurality of standard short sentences, including:

[0025] Sort the multiple sentences according to their positions in the first plain text to obtain a total number of sentences;

[0026] Determine whether the i-th sentence is less than the second length threshold, where i is initially 1;

[0027] If it is less than the second length threshold, the i-th sentence is merged with the i+1-th sentence to form a new sentence, i=i+2;

[0028] If it is not less than the second length threshold, then i=i+1;

[0029] When i is equal to the total number of sentences, determine whether the sentence is greater than the first length threshold;

[0030] If it is not greater than the first length threshold, the sentence is determined to be a standard short sentence;

[0031] If it is greater than the first length threshold, detecting whether the sentence contains an end mark keyword;

[0032] If the end mark keyword is not included, dividing the sentence into standard short sentences based on punctuation marks;

[0033] If the end mark keyword is included, determining whether the number of the end mark keywords is greater than 1;

[0034] If it is not greater than 1, the sentence is divided into standard short sentences based on the end marker keywords and punctuation marks;

[0035] If it is greater than 1, each end-marking keyword is used as a node to divide the sentence into multiple pairs of short sentences;

[0036] Get the length difference between each pair of short sentences, and select the pair of short sentences with the smallest length difference as the standard short sentences.

[0037] Optionally, before the length of the plurality of sentences is normalized based on the first length threshold and the second length threshold to obtain a plurality of standard short sentences, the method further includes:

[0038] constructing a sentence length distribution graph based on the plurality of sentences;

[0039] Obtaining the maximum and minimum values ​​in the sentence length distribution graph;

[0040] Obtaining a median based on the maximum value and the minimum value;

[0041] Drawing a median line in the sentence length distribution graph based on the median, dividing the sentence length distribution graph into an upper area and a lower area;

[0042] Obtaining a first area of ​​the upper region and a second area of ​​the lower region;

[0043] If the first area is equal to the second area, the preset first threshold is determined as the first length threshold, and the preset second threshold is determined as the second length threshold;

[0044] If the first area is not equal to the second area, obtaining an adjustment multiple based on the median, the first area, and the second area;

[0045] Based on the preset first threshold, the preset second threshold, and the adjustment multiple, the first length threshold and the second length threshold are acquired.

[0046] Optionally, the calculation formula of the adjustment multiple is as follows:

[0047]

[0048] Wherein, F is the adjustment multiple; A is the first area; B is the second area; T1 is the preset first threshold; T2 is the preset second threshold; m is the median; A preset positive number.

[0049] Optionally, adding a highlight mark to the first plain text according to the start and end positions to form a text structure with a highlight mark includes:

[0050] According to each obtained start and end position, a pair of inline style tags are added to the first plain text respectively;

[0051] Embed a preset highlight color code in each pair of inline style tags to form a text structure with highlight marks.

[0052] In a second aspect, the embodiment of the present disclosure also provides a system for checking and highlighting text duplication, which adopts the following technical solution:

[0053] A separation module, used for receiving a target text, performing structural analysis on the target text, and separating the original format and the first plain text;

[0054] An acquisition module, configured to acquire a repeated segment between the first plain text and a preset text based on a first length threshold and a second length threshold;

[0055] A positioning module, used to locate the starting and ending positions of the repeated segment in the first plain text;

[0056] An adding module, used for adding a highlight mark to the first plain text according to the start and end positions to form a text structure with a highlight mark;

[0057] An integration module, used for integrating the text structure with the original format to generate a formatted text;

[0058] A display module is used to display the format text.

[0059] In a third aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical solution:

[0060] The computer device comprises:

[0061] at least one processor; and,

[0062] a memory communicatively connected to the at least one processor; wherein,

[0063] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned methods for checking for duplicate text and highlighting it.

[0064] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned methods for checking for duplicate text and highlighting it.

[0065] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0066] The method for checking duplicate text and highlighting provided by the embodiment of the present disclosure can quickly identify the repeated fragments between the target text and the preset text through structured parsing and processing of the plain text. In this process, the first length threshold and the second length threshold are used to effectively reduce the errors or omissions that may exist in the traditional duplicate checking method, thereby improving the accuracy of the duplicate checking. By accurately locating the start and end positions of the repeated fragments in the first plain text and adding highlight marks at these positions, the accuracy of the highlighted display is ensured, and the user can intuitively see which parts are repeated. This visual emphasis makes the duplicate checking result more obvious, which is convenient for the user to quickly identify the repeated content. By separating the original format during the duplicate checking process and integrating it when the formatted text is finally generated, it is ensured that the format of the original text (such as paragraph, font, font size, alignment, etc.) is maintained when the duplicate checking result is displayed, which not only ensures the accuracy of the duplicate checking result, but also improves the user experience. From receiving the target text, parsing, separating, locating the repeated fragments, adding marks to finally generating the formatted text, the whole process is automated, which not only improves the efficiency of the duplicate checking, but also reduces the need for manual intervention.

[0067] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A flowchart of a method for checking for duplicate text and highlighting text provided in an embodiment of the present disclosure;

[0070] Figure 2 A flowchart of a target text separation method provided by an embodiment of the present disclosure;

[0071] Figure 3 A schematic diagram of a flow chart of a method for obtaining repeated segments provided in an embodiment of the present disclosure;

[0072] Figure 4 A schematic diagram of a flow chart of a method for obtaining repeated segments provided in an embodiment of the present disclosure;

[0073] Figure 5 A schematic diagram of a flow chart of a method for dynamically adjusting a length threshold provided in an embodiment of the present disclosure;

[0074] Figure 6 A logic block diagram of a method for checking for duplicate text and highlighting text provided in an embodiment of the present disclosure;

[0075] Figure 7 A functional block diagram of a system for checking and highlighting text duplication provided in an embodiment of the present disclosure;

[0076] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0077] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0078] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0079] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0080] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0081] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0082] Reference Figure 1 The present invention provides a method for checking and highlighting text duplication, comprising the following steps:

[0083] S1: receiving the target text, performing structural analysis on the target text, and separating the original format and the first plain text;

[0084] S2: based on the first length threshold and the second length threshold, obtaining a repeated segment between the first plain text and the preset text;

[0085] S3: locate the start and end positions of the repeated segment in the first plain text;

[0086] S4: adding highlight marks to the first plain text according to the start and end positions to form a text structure with highlight marks;

[0087] S5: Integrate the text structure with the original format to generate formatted text;

[0088] S6: Display the formatted text.

[0089] The disclosed method of checking duplicate text and highlighting it can quickly identify the repeated fragments between the target text and the preset text through structured parsing and processing of the plain text. In this process, the first length threshold and the second length threshold are used to effectively reduce the errors or omissions that may exist in the traditional duplicate checking method, thereby improving the accuracy of the duplicate checking. By accurately locating the start and end positions of the repeated fragments in the first plain text and adding highlight marks at these positions, the accuracy of the highlight display is ensured, and the user can intuitively see which parts are repeated. This visual emphasis makes the duplicate checking result more obvious, which is convenient for the user to quickly identify the repeated content. By separating the original format during the duplicate checking process and integrating it when the formatted text is finally generated, it is ensured that the format of the original text (such as paragraph, font, font size, alignment, etc.) is maintained when the duplicate checking result is displayed, which not only ensures the accuracy of the duplicate checking result, but also avoids interfering with the user's document editing habits, thereby improving the user experience. From receiving the target text, parsing, separating, locating the repeated fragments, adding marks to finally generating the formatted text, the whole process is automated, which not only improves the efficiency of the duplicate checking, but also reduces the need for manual intervention. The entire duplicate checking process is designed to be simple and clear. Users only need to provide the target text, and the system will automatically complete all subsequent steps, making it easy for non-technical users to use.

[0090] In summary, this method not only improves the efficiency and accuracy of text duplication checking, but also helps users better understand and deal with duplicate content through visualization, while ensuring that the original format of the target text is not destroyed during the duplication checking process, which is especially important for documents that need to maintain a specific format (such as academic papers, reports, etc.).

[0091] In S1, refer to Figure 2 The flowchart of the target text separation method shown in the figure, "receiving the target text, performing structural analysis on the target text, and separating the original format and the first plain text" includes the following steps:

[0092] S11: receiving the target text through the interface service and performing a security check on the target text; if the security check passes, executing S12; if the security check fails, executing S13;

[0093] S12: using a preset separation tool to convert the text content in the target text into a first plain text, and extracting the original format from the target text and storing it in a database;

[0094] S13: Transmitting text optimization prompts to the sending end.

[0095] In S11-S13, in order to ensure the stability and availability of the business service system, the traditional business service system is split into a business system (Business System) and a similarity checker system (Similarity Checker), where the business system provides an interface service. When the user terminal device sends a target text, the target text is received through the interface service, where the target text is a large text file containing an html editing format.

[0096] The business system sets a duplicate check task for the target text. During the execution of the duplicate check task, the target text is firstly subjected to a security check, and the duplicate check task is adjusted from the initial state to the state of being checked. The security check includes file type check, sandbox environment test, etc. If the security check fails, a text optimization prompt is generated based on the test result and sent to the user terminal device. The user adjusts the target text according to the text optimization prompt and resubmits the adjusted text. If the check passes, the duplicate check task is adjusted from the state of being checked to the state of being checked successfully, and a preset separation tool (such as the org.jsoup open source tool) is called to perform structural analysis on the target text, separating the text content and the original format of the target text, wherein the original format is the structure, style and format information of the target text, including but not limited to the text format, paragraph format and layout. The separated text content is converted into plain text without any format and markup. In order to facilitate subsequent distinction, the plain text separated here is called the first plain text. The first plain text is sent to the duplicate check system, the separated original format is stored in the database, and the duplicate check task is adjusted from the state of being checked successfully to the state of being checked for duplicates.

[0097] The duplicate checking system does not provide interface services. It scans through scheduled tasks to see whether there are duplicate checking tasks in the pending duplication checking state, i.e., tasks to be checked for duplicates. When tasks to be checked for duplicates are scanned, the tasks to be checked for duplicates are executed (i.e., the first plain text is processed), and the duplicate checking tasks are adjusted from the pending duplication checking state to the checking for duplicates in progress state.

[0098] In S2, refer to Figure 3 The flowchart of the method for obtaining repeated segments shown in the embodiment of the present invention, "obtaining repeated segments between the first plain text and the preset text based on the first length threshold and the second length threshold" includes the following steps:

[0099] S21: performing coarse-grained semantic segmentation on the first plain text to obtain multiple semantic segments;

[0100] S22: Perform fine-grained semantic segmentation on multiple semantic segments to obtain multiple sentences;

[0101] S23: Standardizing the lengths of the plurality of sentences based on the first length threshold and the second length threshold to obtain a plurality of standard short sentences;

[0102] S24: extracting content from the preset text to obtain a second plain text;

[0103] S25: Divide the second plain text into multiple segments, and obtain the similarity between the standard short sentence and each segment;

[0104] S26: Determine whether the maximum similarity is less than a preset similarity threshold; if not, execute S27; if yes, execute S29;

[0105] S27: Determine whether the number of consecutively repeated words in the standard short sentence is less than a preset word count threshold; if not, execute S28; if yes, execute S29;

[0106] S28: Determine the standard short sentence as a repeated segment;

[0107] S29: Determine the standard short sentence as a non-repetitive segment.

[0108] In S21, in one possible implementation, the duplicate checking system preprocesses the first plain text, and the preprocessing includes removing redundant spaces, standardizing punctuation marks, etc., and extracting semantic features of the preprocessed first plain text, such as keywords, topics, paragraph markers, etc. The extracted semantic features can convert the text content into a vector representation, which is used to represent the semantic information of the text and provide a basis for subsequent semantic similarity calculations. The semantic similarity between adjacent segments in the first plain text is calculated. If the semantic similarity is not greater than the preset semantic similarity threshold, it is determined that the adjacent segments do not belong to the same semantic unit. If the semantic similarity is greater than the preset semantic similarity threshold, it is determined that the adjacent segments belong to the same semantic unit, and the adjacent segments are merged to form a new segment. Then, based on the new segment, the semantic similarity is recalculated until there are no adjacent segments greater than the preset semantic similarity threshold, and the obtained multiple segments are determined as multiple semantic segments.

[0109] In another possible implementation, a machine learning model, such as a classifier, is trained to identify semantic boundaries in the text and divide the first plain text into a plurality of semantic segments.

[0110] In S22, sentence boundaries in the semantic segments are identified using natural language processing (NLP) techniques (eg, sentence segmentation, machine learning models, etc.).

[0111] In S23, refer to Figure 4 The flow chart of the length standardization method shown, "Based on the first length threshold and the second length threshold, length standardizing multiple sentences to obtain multiple standard short sentences" includes the following steps:

[0112] S231: Sort the multiple sentences according to their positions in the first plain text to obtain the total number of sentences.

[0113] S232: Determine whether the i-th sentence is less than a second length threshold, where i is initially 1; if so, execute S233; if not, execute S234.

[0114] S233: Merge the i-th sentence with the i+1-th sentence to form a new sentence, i=i+2.

[0115] S234:i=i+1.

[0116] S235: Determine whether i is equal to the total number of sentences; if so, execute S236; if not, return to S232.

[0117] When i is equal to the total number of sentences, it means that adjacent sentences with shorter lengths have been merged. In subsequent duplicate checking, the impact of sentences that are too short on the accuracy of duplicate checking is effectively reduced.

[0118] S236: Determine whether the sentence is greater than the first length threshold; if not, execute S237; if so, execute S238.

[0119] S237: Determine the sentence as a standard short sentence.

[0120] S238: Check whether the sentence contains the end mark keyword; if not, execute S239; if so, execute S240.

[0121] Among them, the end mark keyword refers to the vocabulary that marks the end of a semantic unit, which usually appears at the beginning of a semantic unit and marks the end of the previous semantic unit. The end mark keywords include at least one of but, however, nevertheless, therefore, so, due to, finally, ultimately, in short, in short, subsequently, then, and next.

[0122] S239: Divide sentences into standard short sentences based on punctuation marks.

[0123] Detect the middle words and all punctuation marks of the sentence, obtain the distance between each punctuation mark and the middle words, and divide the sentence into two standard short sentences with the punctuation mark closest to the middle words as the node. The middle words refer to the words in the middle of the sentence. If the sentence length is odd, the middle words are 1 character, and if the sentence length is even, the middle words are 2 characters.

[0124] If the middle word is 1 character, the distance Dqp between the punctuation mark and the middle word is calculated as follows:

[0125] Dqp=|dq-dp|

[0126] Where q represents punctuation marks; p represents the middle word; dq represents the position of the punctuation marks in the sentence; and dp represents the position of the middle word in the sentence.

[0127] If the middle word is 2 characters, the distance Dqp between the punctuation mark and the middle word is calculated as follows:

[0128]

[0129] In the formula, is the average of the positions of the two characters in the middle word in the sentence.

[0130] S240: Determine whether the number of end mark keywords is greater than 1; if not, execute S241; if so, execute S242.

[0131] S241: Divide the sentence into standard short sentences based on the end marker keywords and punctuation marks.

[0132] Taking the end mark keyword as a node, the sentence is divided into a pair of short sentences. If the length difference between a pair of short sentences is not greater than the preset length difference threshold, the pair of short sentences are judged as standard short sentences; if the length difference between a pair of short sentences is greater than the preset length difference threshold, refer to the specific steps of S239 and divide the sentences into standard short sentences based on punctuation marks.

[0133] S242: Taking each end-marking keyword as a node, the sentence is divided into a plurality of pairs of short sentences;

[0134] The content of the sentence before the end marker keyword is taken as a short sentence, and the content after the end marker keyword (including the end marker keyword itself) is taken as another short sentence, thereby obtaining a pair of short sentences.

[0135] S243: Obtain the length difference between each pair of short sentences, and select a pair of short sentences with the smallest length difference as the standard short sentences.

[0136] Sentences that are too long are divided into standard short sentences by the first length threshold, and the influence of sentences that are too long on the accuracy of the duplicate check is effectively reduced during the subsequent duplicate check. The present disclosure ultimately detects repeated segments by the number of consecutively repeated words in the standard short sentences. The length of the standard short sentences themselves has a great influence on the accuracy of the final detection. Therefore, the present disclosure combines the first length threshold and the second length threshold so that the length of the standard short sentences is within an appropriate range, which can effectively improve the accuracy of the duplicate check.

[0137] Furthermore, due to the different types of target texts and the personal habits of users, the overall sentence lengths of different target texts are inconsistent, that is, some target texts may use more long sentences, while others may use more short sentences. Therefore, before standardizing the lengths of multiple sentences based on the first length threshold and the second length threshold to obtain multiple standard short sentences, the first length threshold and the second length threshold are dynamically adjusted in combination with the sentence length distribution of the target text. This can specifically adapt to target texts with different sentence length distributions, making the restriction on the length of standard short sentences more precise.

[0138] Reference Figure 5 The flowchart of the method for dynamically adjusting the length threshold is shown, and the method for dynamically adjusting the first length threshold and the second length threshold includes the following steps:

[0139] S244: construct a sentence length distribution graph based on the multiple sentences, and obtain the maximum value and the minimum value in the sentence length distribution graph.

[0140] Get the length of each sentence, set a sequence number for each sentence according to the order of the sentence in the first plain text, and construct an xoy coordinate system. The x-axis in the xoy coordinate system represents the sequence number of the sentence, and the y-axis represents the length of the sentence. Draw a point cloud in the sentence length distribution diagram based on the sequence number and length of the sentence. The expression of each point is (x, y). Connect each point in order to form a curve. At this point, the sentence length distribution diagram is completed.

[0141] S245: Obtain a median based on the maximum value and the minimum value, and draw a median line in the sentence length distribution graph based on the median.

[0142] Identify the maximum value y in the sentence length distribution graph max and the minimum value y min , find the average of the two, that is, the median m, m= (y max +y min ) / 2, draw the median line in the sentence length distribution diagram based on the median, and the expression of the median line is: y=m.

[0143] S246: The median line divides the sentence length distribution graph into an upper region and a lower region, and obtains a first area of ​​the upper region and a second area of ​​the lower region.

[0144] The upper area represents the part where the sentence length is greater than the median, and the lower area represents the part where the sentence length is less than the median. Mathematical statistical methods are used to calculate the area of ​​the upper area and the area of ​​the lower area. For ease of distinction, the area of ​​the upper area is called the first area, and the area of ​​the lower area is called the second area.

[0145] S247: Determine whether the first area is equal to the second area; if so, execute S248; if not, execute S249.

[0146] S248: Determine the preset first threshold as the first length threshold, and determine the preset second threshold as the second length threshold.

[0147] When the first area is equal to the second area, it means that the sentence length distribution of the first plain text is uniform. The preset first threshold is directly used as the first length threshold, and the preset second threshold is used as the second length threshold. The preset first threshold and the preset second threshold are obtained through a large number of experiments and are suitable for the case where the sentence length distribution is uniform. In actual applications, a user feedback channel is set to fine-tune the preset first threshold and the preset second threshold through user feedback.

[0148] S249: Based on the median, the first area, and the second area, an adjustment multiple is obtained; based on the preset first threshold, the preset second threshold, and the adjustment multiple, a first length threshold and a second length threshold are obtained.

[0149] The calculation formula for the adjustment multiple is as follows:

[0150]

[0151] Wherein, F is the adjustment factor; A is the first area; B is the second area; T1 is the preset first threshold; T2 is the preset second threshold; is a preset positive number, and is set in this disclosure 1×10 -6 , to avoid the situation where the denominator is 0.

[0152] In S24-S28, if the preset text is a historically processed text, the business system directly retrieves the second plain text of the preset text from the database; if the preset text is a new text, the business system calls the preset separation tool to extract the content of the preset text, obtains the second plain text, sends the second plain text to the duplicate checking system, and stores the second plain text in the database.

[0153] The duplicate checking system divides the second plain text into multiple segments, and uses a preset similarity algorithm, such as cosine similarity, Jaccard similarity, Google open source algorithm diff-match-patch, etc., to obtain the similarity between the standard short sentence and each segment, and obtain the maximum similarity therein. If the maximum similarity is not less than the preset similarity threshold, then the segment most similar to the standard short sentence is obtained based on the maximum similarity, and the number of consecutive repeated words between the standard short sentence and its most similar segment is obtained. If the number of consecutive repeated words of the standard short sentence is not less than the preset word count threshold, the standard short sentence is determined to be a repeated segment. If the maximum similarity is less than the preset similarity threshold or the number of consecutive repeated words of the standard short sentence is less than the preset word count threshold, the standard short sentence is determined to be a non-repeated segment. Repeat the above steps, detect each standard short sentence, and filter out all repeated segments.

[0154] Optionally, different business scenarios have different tolerances for duplicate content. Therefore, the word count threshold is set based on the current business scenario. For example, the word count threshold is set to 5 in the field of news duplicate checking, and the word count threshold is set to 8 in the field of paper duplicate checking. This method of dynamically adjusting the word count threshold according to the current business scenario can more accurately identify and determine duplicate fragments, reduce misjudgments and missed judgments, provide duplicate checking results that are more in line with user needs, and improve user satisfaction and trust.

[0155] When the amount of tasks to be checked for duplicate content is greater than the preset task volume threshold, the CPU utilization rate of the duplicate checking system is elastically expanded. For example, the number of CPU cores, content, and servers can be increased, so that more tasks to be checked for duplicate content can be processed at the same time, reducing the waiting time of the business.

[0156] After the duplicate check is completed, the duplicate check results will be written into the database, and the duplicate check system will adjust the duplicate check status of the duplicate check task to the duplicate check completed status.

[0157] For repeated duplicate checking requests, a cache mechanism is used to store the processed text comparison results. When the duplicate checking system receives the same duplicate checking request again, it can directly obtain the results from the cache without having to check again, which can greatly reduce the processing time.

[0158] In S3, the starting position and ending position of each repeated segment in the first plain text are recorded. Character index or word index can be used to record the position information. The position information of all repeated segments is stored in a data structure, such as an array or a list. Each position information entry contains the starting index and ending index of the repeated segment for subsequent use.

[0159] In S4, in the new text structure, for each highlighted part, the business system will generate a corresponding CSS style or other formatting instructions. These instructions will act on the repeated content when displayed, so that it can be displayed in the specified highlighting style while maintaining the original format.

[0160] Specifically, for each start and end position, add a pair of inline style tags to the first plain text. The inline style tags include the start tag and the end tag , a preset highlight color code is embedded in each pair of inline style tags to form a text structure with highlight marks, where the highlight mark can be a mark such as a special color, bold, underline or background color that can distinguish between repeated segments and non-repeated segments.

[0161] For example, in business scenario A, the duplicate checking system processes HTML paragraphs containing content (such as abcdefghljk ), detects and identifies the repeated content "bcdef", and the repeated content meets the preset word count threshold (for example, greater than or equal to 5 characters). The highlighted document structure generated by the business system is: a <span style="”" color: rgb(255,0, 1);”> bcdef g , where the "bcdef" part is highlighted.

[0162] In S5, refer to Figure 6 The logic diagram of the method for checking and highlighting duplicate text is shown. The business system sends the original format and text structure to the formatter. The formatter is a tool used to convert text or data into a specific format. In text processing, it can convert plain text into a document with a specific style and layout, such as applying different fonts, sizes, colors, and paragraph formats. It can also convert data into a format suitable for a specific purpose. The formatter is used to integrate the text structure with the original format, and the integrated text is exported as a CSV, JSON, or XML file according to the user's specifications, thereby generating formatted text.

[0163] In S6, the formatter sends the formatted text to the display through the interface service of the business system. The display refers to a component or interface used to display or show information. In the context of text processing, the display can be a user interface that displays the processed formatted text to the user in a readable and visually attractive way. The display can also be a web page, desktop application, mobile application or any other platform that can present text and multimedia content. Design a user-controllable highlight switch so that users can freely switch or compare between viewing the original target text and viewing the formatted text to improve the user experience. Moreover, the final formatted text can be directly displayed to the user without manual processing or conversion, which improves the user experience and convenience of operation.

[0164] This method uses double-layer rendering or overlay rendering technology to divide the target text into a bottom layer and an upper layer. The bottom layer retains the original format of the target text and is used to restore the text according to the original format edited by the user after the duplicate check is completed. The upper layer adds a duplicate check highlight effect separately to ensure that the highlight mark and the original format set by the user do not interfere with each other. The formatted text finally generated only adds the highlight mark part. Tags and their style attributes, which not only retains the original format but also does not change the original format, improving user readability.

[0165] Reference Figure 7 The present disclosure provides a system for checking and highlighting text duplication, comprising:

[0166] The separation module 101 is used to receive the target text, perform structural analysis on the target text, and separate the original format and the first plain text;

[0167] An acquisition module 102 is used to acquire a repeated segment between the first plain text and the preset text based on a first length threshold and a second length threshold;

[0168] A positioning module 103, used to locate the start and end positions of the repeated segment in the first plain text;

[0169] An adding module 104 is used to add a highlight mark to the first plain text according to the start and end positions to form a text structure with a highlight mark;

[0170] An integration module 105, for integrating the text structure with the original format to generate a formatted text;

[0171] The display module 106 is used to display the formatted text.

[0172] The various variations and specific examples of the method for checking for duplicate text and highlighting provided above are also applicable to the system for checking for duplicate text and highlighting provided in the present invention. Through the above detailed description of the method for checking for duplicate text and highlighting, those skilled in the art can clearly understand the implementation method of the system for checking for duplicate text and highlighting. For the sake of brevity of the specification, it will not be described in detail here.

[0173] The computer device according to the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0174] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device performs all or part of the steps of the method for checking and highlighting text duplication in the aforementioned embodiments of the present disclosure.

[0175] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0176] like Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown, which is a schematic diagram of the structure of a computer device suitable for implementing the embodiment of the present disclosure. Figure 8 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0177] like Figure 8 As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0178] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 8 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0179] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the method for checking for duplicate text and highlighting the embodiment of the present disclosure are executed.

[0180] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0181] According to the computer-readable storage medium of the embodiment of the present disclosure, non-transitory computer-readable instructions are stored thereon. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the method for checking for duplicate text and highlighting the text in the above-mentioned embodiments of the present disclosure are executed.

[0182] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0183] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0184] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0185] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be recognized by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, meaning "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0186] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a disjunctive list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0187] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0188] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0189] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0190] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for checking and highlighting text duplication, characterized in that: include: Receiving a target text, performing structural analysis on the target text, and separating an original format and a first plain text; Performing coarse-grained semantic segmentation on the first plain text to obtain multiple semantic segments; Performing fine-grained semantic division on the multiple semantic segments to obtain multiple sentences; Based on a first length threshold and a second length threshold, length-standardizing the plurality of sentences to obtain a plurality of standard short sentences; Before the length of the plurality of sentences is normalized based on the first length threshold and the second length threshold to obtain a plurality of standard short sentences, the method further includes: constructing a sentence length distribution graph based on the plurality of sentences; Obtaining the maximum and minimum values ​​in the sentence length distribution graph; Obtaining a median based on the maximum value and the minimum value; Drawing a median line in the sentence length distribution graph based on the median, dividing the sentence length distribution graph into an upper area and a lower area; Obtaining a first area of ​​the upper region and a second area of ​​the lower region; If the first area is equal to the second area, the preset first threshold is determined as the first length threshold, and the preset second threshold is determined as the second length threshold; If the first area is not equal to the second area, obtaining an adjustment multiple based on the median, the first area, and the second area; Based on the preset first threshold, the preset second threshold and the adjustment multiple, acquiring the first length threshold and the second length threshold; Extracting the content of the preset text to obtain a second plain text; dividing the second plain text into a plurality of segments; Obtaining the similarity between the standard short sentence and each paragraph; When the maximum similarity is not less than a preset similarity threshold, obtaining the number of consecutively repeated words of the standard short sentence; When the number of consecutively repeated words is not less than a preset word count threshold, the standard short sentence is determined to be a repeated segment; Locating the start and end positions of the repeated segment in the first plain text; Adding highlight marks to the first plain text according to the start and end positions to form a text structure with highlight marks; Integrate the text structure with the original format to generate formatted text; The formatted text is displayed.

2. The method for checking and highlighting text duplication according to claim 1, characterized in that: The receiving of the target text, performing structural analysis on the target text, and separating the original format and the first plain text includes: receiving the target text through an interface service, and performing a security check on the target text; If the security check passes, a preset separation tool is used to convert the text content in the target text into the first plain text, and the original format is extracted from the target text and stored in a database; If the security check fails, a text optimization prompt is transmitted to the user terminal device.

3. The method for checking and highlighting text duplication according to claim 1, characterized in that: The length of the plurality of sentences is normalized based on the first length threshold and the second length threshold to obtain a plurality of standard short sentences, including: Sort the multiple sentences according to their positions in the first plain text to obtain a total number of sentences; Determine whether the i-th sentence is less than the second length threshold, where i is initially 1; If it is less than the second length threshold, the i-th sentence is merged with the i+1-th sentence to form a new sentence, i=i+2; If it is not less than the second length threshold, then i=i+1; When i is equal to the total number of sentences, determine whether the sentence is greater than the first length threshold; If it is not greater than the first length threshold, the sentence is determined to be a standard short sentence; If it is greater than the first length threshold, detecting whether the sentence contains an end mark keyword; If the end mark keyword is not included, dividing the sentence into standard short sentences based on punctuation marks; If the end mark keyword is included, determining whether the number of the end mark keywords is greater than 1; If it is not greater than 1, the sentence is divided into standard short sentences based on the end marker keywords and punctuation marks; If it is greater than 1, each end-marking keyword is used as a node to divide the sentence into multiple pairs of short sentences; Get the length difference between each pair of short sentences, and select the pair of short sentences with the smallest length difference as the standard short sentences.

4. The method for checking for duplicate text and highlighting it according to claim 1, characterized in that: The calculation formula of the adjustment multiple is as follows: Among them, F is the adjustment factor; A is the first area; B is the second area; T1 is the preset first threshold; T2 is the preset second threshold; m is the median; ϵ is a preset positive number.

5. The method for checking for duplicate text and highlighting it according to claim 1, characterized in that: The step of adding a highlight mark to the first plain text according to the start and end positions to form a text structure with a highlight mark includes: According to each obtained start and end position, a pair of inline style tags are added to the first plain text respectively; Embed a preset highlight color code in each pair of inline style tags to form a text structure with highlight marks.

6. The method for checking for duplicate text and highlighting text according to claim 5, characterized in that: The pair of inline style tags includes a start tag and the end tag .

7. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for checking for duplicate text and highlighting it as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the method for checking for duplicate text and highlighting it as described in any one of claims 1-6.

9. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images