Method and device for searching difference between documents based on big data

Abstract

The invention discloses a method and a device for searching differences among documents based on big data, and relates to the technical field of electric digital data processing. The method comprises the steps of obtaining a to-be-searched document after format analysis, and preprocessing text content in the to-be-searched document; performing difference recognition on the preprocessed text content based on a text comparison algorithm, obtaining a complexity evaluation index, determining a difference recognition result according to the complexity evaluation index, and visualizing the difference recognition result through highlight display; adjusting parameters of a text comparison algorithm according to the highlighted visualization result, and performing difference recognition again according to the adjusted text comparison algorithm to obtain a document difference score; the document difference score is used for measuring the difference degree between the to-be-searched documents. By adopting the method and the device, the accuracy of finding the difference content between the documents is improved, and the problem that the difference content between the documents is not sufficiently found in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a method and device for searching differences between documents based on big data. Background Art

[0002] With the rapid development of information technology and the widespread application of the Internet, the amount of global data has exploded. Enterprises, government agencies, scientific research institutions and other fields have accumulated massive amounts of document data. These data contain rich information, but they also bring challenges to management and analysis. How to quickly and accurately find differences between documents in massive data has become an urgent problem to be solved. Secondly, in the era of information explosion, users' demand for accurate information is increasing. Comparing differences between documents can help users quickly locate the change points of information, so as to better understand and use this information. Big data technology provides strong technical support for finding differences between documents and can efficiently process massive data. The method and system for finding differences between documents based on big data use these technologies to achieve rapid comparison and difference identification of document data.

[0003] The existing method of finding differences between documents uses a neural network model to identify key elements in the document, and then allows the user to compare the key elements in the two documents to identify the difference through the "compare" function in the Word document processing software, and simultaneously displays the document content to visualize the content of the difference, thereby realizing the automatic identification of the difference between the document contents. However, in the prior art, the neural network model is limited by the complexity and diversity of the document data, resulting in omissions and misjudgments in the neural network model during the difference identification process. Secondly, when there are a lot of blank spaces or chaotic formats in the document content, the comparison algorithm in the Word document processing software cannot accurately process the text differences therein, resulting in the complete identification of the differences in the document content, and there is a problem of insufficient search for the difference content between documents. Summary of the invention

[0004] In order to solve the technical problem of insufficient search for differences between documents in the prior art, an embodiment of the present invention provides a method and device for searching for differences between documents based on big data. The technical solution is as follows:

[0005] In one aspect, a method for finding differences between documents based on big data is provided, the method being implemented by a device for finding differences between documents based on big data, the method comprising:

[0006] S1, obtaining a document to be searched after format analysis, and preprocessing the text content in the document to be searched;

[0007] S2. Based on the text comparison algorithm, the preprocessed text content is identified for differences, and a complexity evaluation index is obtained. The difference identification result is determined according to the complexity evaluation index, and the difference identification result is visualized by highlighting;

[0008] S3. According to the highlighted visualization results, adjust the parameters of the text comparison algorithm, re-perform difference identification according to the adjusted text comparison algorithm, and obtain a document difference score; the document difference score is used to measure the degree of difference between the documents to be found.

[0009] On the other hand, a device for finding differences between documents based on big data is provided, and the device is applied to a method for finding differences between documents based on big data, and the device comprises:

[0010] A preprocessing module is used to obtain the document to be searched after format parsing, and preprocess the text content in the document to be searched;

[0011] A determination module is used to identify differences in the preprocessed text content based on a text comparison algorithm, obtain a complexity evaluation index, determine a difference identification result according to the complexity evaluation index, and visualize the difference identification result by highlighting;

[0012] The acquisition module is used to adjust the parameters of the text comparison algorithm according to the highlighted visualization results, re-identify the differences according to the adjusted text comparison algorithm, and obtain the document difference score; the document difference score is used to measure the degree of difference between the documents to be found.

[0013] On the other hand, a device for searching for differences between documents based on big data is provided, the device comprising: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the methods for searching for differences between documents based on big data is implemented.

[0014] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for finding differences between documents based on big data.

[0015] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0016] The above-mentioned scheme is a method and system for finding differences between documents based on big data. It preprocesses the text content in the document to be found after format analysis, and identifies the differences of the preprocessed text content based on the text comparison algorithm. Then, the difference identification process is monitored in real time to obtain complexity evaluation indicators. At the same time, the results of the difference identification are visualized by highlighting. Finally, the parameters of the text comparison algorithm are adjusted according to the highlighted visualization results and the difference identification is re-performed to obtain the document difference score, thereby improving the accuracy of finding the difference content between documents and solving the problem of insufficient finding of the difference content between documents in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flow chart of a method for finding differences between documents based on big data provided by an embodiment of the present invention;

[0019] Figure 2 A three-dimensional coordinate analysis diagram of a recognition efficiency evaluation value provided by an embodiment of the present invention;

[0020] Figure 3 It is a block diagram of a device for searching differences between documents based on big data provided by an embodiment of the present invention;

[0021] Figure 4 It is a structural diagram of a device for searching differences between documents based on big data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0023] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0024] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0025] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0026] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0027] An embodiment of the present invention provides a method for finding differences between documents based on big data. The method can be implemented by a device for finding differences between documents based on big data. The device for finding differences between documents based on big data can be a terminal or a server. The present invention aims to solve the problem that existing neural network models are limited by the complexity and diversity of document data, which leads to omissions and misjudgments in the difference identification process of the neural network model. Secondly, when there are a lot of blank spaces or chaotic formats in the document content, the comparison algorithm in the Word document processing software cannot accurately process the text differences therein, which leads to the problem that the difference content between documents cannot be fully found. A method and system for finding differences between documents based on big data are provided, and the accuracy of finding the difference content between documents is effectively improved. Figure 1 The flowchart of the method for finding differences between documents based on big data is shown. The processing flow of the method may include the following steps:

[0028] S1. Obtain the document to be searched after format parsing, and pre-process the text content in the document to be searched.

[0029] In a feasible implementation, format analysis means analyzing the format of the document to be searched, and preprocessing includes removing document format information, standardizing character encoding and word segmentation. The document format information includes fonts, colors and pictures. When the document to be searched is a video or a picture, the corresponding text content is the text content in the video or picture.

[0030] S2. Based on the text comparison algorithm, the difference of the preprocessed text content is identified, and the complexity evaluation index is obtained. The difference identification result is determined according to the complexity evaluation index, and the difference identification result is visualized by highlighting.

[0031] Among them, the text comparison algorithm is used to compare and analyze the preprocessed text content.

[0032] Optionally, the text comparison algorithm based on S2 performs difference identification on the preprocessed text content, including the following steps S21-S22:

[0033] S21, converting the text content after removing stop words into a numerical vector and performing a morphological normalization operation to obtain common subsequences in the documents to be searched in the current preset time period and the previous preset time period, the morphological normalization operation including temporal normalization and singular-plural normalization.

[0034] S22. Obtain the edit distance in the common subsequence using a dynamic programming algorithm, and record the differences of corresponding positions in the common subsequence in real time according to the edit distance.

[0035] The edit distance represents the minimum number of edit operations required to convert text content within a preset time period into text content within the previous preset time period.

[0036] In a feasible implementation, the text comparison algorithm includes the Longest Common Subsequence (LCS) algorithm and the Diff algorithm. In this embodiment, the differences of the common subsequences in the document to be searched are recorded in real time through the Longest Common Subsequence algorithm. Stop words usually refer to words that do not affect the meaning of the text content (such as "de", "shi" in Chinese). Word segmentation is to split the text content into words or phrases, which helps to improve the accuracy of difference recognition. Through stop word processing and word segmentation processing, the words that do not affect the content meaning in the document to be searched are excluded, reducing noise interference and helping to more accurately identify the differences in the text. Tense normalization is to convert the verbs in the text content into the same tense (such as the simple present tense), which usually uses the part-of-speech tagging (POS tagging) and verb tense conversion functions in the Natural Language Processing (NLP) library. Singular-plural normalization is to convert the nouns in the text content into singular or plural forms. To obtain the common subsequences in the document to be searched within a preset time period and the previous preset time period, the dynamic programming algorithm is usually used to compare the word sequences of two adjacent documents to find the longest common subsequence in the text content. The Levenshtein Distance (LD) is a measure used to measure the difference between two sequences, that is, the minimum number of single-character editing (insertion, deletion, or replacement) operations required to convert one string into another string, which helps to identify the differences in the text content at different positions in the document to be searched. The method of this embodiment supports real-time recording and analysis of the differences in the document to be searched, can timely feedback the changes in the text content, and provide timely and accurate decision-making support for relevant personnel. This method is not only applicable to the field of text difference recognition and analysis, but also can be extended to other scenarios that require text comparison and processing, such as document version control and information retrieval, achieving an improvement in the efficiency and accuracy of difference recognition.

[0037] Optionally, after the difference recognition of the preprocessed text content based on the text comparison algorithm, the method further includes:

[0038] Obtain the recognition efficiency evaluation value according to the following formula (1), and the recognition efficiency evaluation value is used to measure the difference recognition efficiency of the text content in the document to be searched within the current preset time period.

[0039]

[0040] Where r is the number of the document to be searched, r = 1, 2,..., R, R is the total number of documents to be searched, t is the number of the preset time period, t = 1, 2,..., T, T is the total number of preset time periods, and e is the natural constant, YI r.tA1 represents the recognition efficiency evaluation value of the rth document to be searched in the tth preset time period, r.t A2 represents the actual number of preprocessed characters of the rth document to be searched in the tth preset time period. r.t represents the actual number of difference recognition characters of the rth document to be searched in the tth preset time period, ΔA1 r.t represents the number of reference preprocessed characters of the rth document to be searched within the tth preset time period, ΔA2 r.t T1 represents the number of reference difference recognition characters of the rth document to be searched within the tth preset time period, r.t represents the actual preprocessing time of the rth document to be searched in the tth preset time period, T2 r.t Indicates the actual difference recognition time of the rth document to be searched within the tth preset time period, ΔT1 r.t represents the reference preprocessing time of the rth document to be searched in the tth preset time period, ΔT2 r.t Indicates the reference difference recognition time of the rth document to be searched within the tth preset time period.

[0041] In a feasible implementation manner, the actual number of difference recognition characters refers to the number of difference characters obtained during the difference recognition process, which is usually the difference between the number of difference content characters within a preset time period and the number of difference content characters within a previous preset time period, wherein difference recognition represents the process of identifying the difference content between the text content to be searched in the preset time period and the text content to be searched in the previous preset time period, and the time difference recognition time refers to the time used for difference recognition. It should be noted that when there is no previous preset time period (that is, the current preset time period is numbered 1), the original document of the document to be searched is compared; the reference preprocessing character number, the reference difference recognition character number, the reference preprocessing time and the reference difference recognition time are all not 0, wherein the reference preprocessing character number is represented by the result of summing and averaging the number of historical preprocessing text content characters within a unit time period in a preset database, the reference difference recognition character number is represented by the result of summing and averaging the number of historical difference recognition text content characters within a unit time period in the preset database, the reference preprocessing time is represented by the result of summing and averaging the number of historical preprocessing times of the number of text content characters in the preset database, and the reference difference recognition time is represented by the result of summing and averaging the number of historical difference recognition times of the number of text content characters in the preset database.

[0042] In this embodiment, the recognition efficiency evaluation value increases with the increase of the actual number of preprocessed characters and the actual number of difference recognized characters, and decreases with the increase of the actual preprocessing time and the actual difference recognition time. It should be noted that the increase of the actual number of preprocessed characters indicates that more text content can be processed within the preset time period, while the reduction of the actual preprocessing time directly reflects the improvement of the preprocessing speed, which helps to speed up the document or data preparation stage and gain more time for subsequent difference recognition and analysis work; the increase of the actual number of difference recognized characters indicates that more difference content can be recognized within the preset time period, while the reduction of the actual difference recognition time indicates that the speed of difference recognition is faster, which helps to locate the differences between documents more quickly and accurately and improve the overall work efficiency. By comprehensively considering the two dimensions of the number of characters and time to evaluate the recognition efficiency, this embodiment can more comprehensively reflect the performance of the recognition process, which helps to more accurately evaluate the reliability of the recognition results and reduce subsequent problems caused by recognition errors or omissions.

[0043] To simplify the analysis, we define Where W1 r.t W2 represents the coefficient of the number of characters recognized in the rth document to be searched within the tth preset time period, r.t It represents the recognition time coefficient of the rth document to be searched in the tth preset time period. The simplified calculation formula of the recognition efficiency evaluation value is: like Figure 2 As shown, it is a three-dimensional coordinate analysis diagram of the recognition efficiency evaluation value provided by an embodiment of the present invention. It can be seen from the figure that the recognition efficiency evaluation value increases with the increase of the recognition character number coefficient, and decreases with the increase of the recognition time coefficient, but the decrease amplitude of the recognition time coefficient is smaller than the increase amplitude of the recognition character number coefficient, indicating that in the range of 0 to 1, the influence of the recognition character number coefficient on the recognition efficiency evaluation value is greater than the influence of the recognition time coefficient on the recognition efficiency evaluation value.

[0044] It should be understood that the coefficient of the number of recognized characters also indirectly affects the value of the recognition time coefficient. When the number of characters contained in a document increases, the corresponding complexity of the document content increases, that is, it takes longer to process and recognize. Therefore, when the coefficient of the number of recognized characters (that is, the number of characters in the document) increases, the recognition time coefficient (that is, the time required to process these characters) will also increase accordingly. By considering the indirect impact of the coefficient of the number of recognized characters on the recognition time coefficient, it is helpful to more comprehensively reflect the recognition accuracy and efficiency between document contents, and improve the accuracy and reliability of obtaining the recognition efficiency evaluation value, thereby achieving an improvement in the accuracy of finding the difference content between documents, and effectively solving the problem of insufficient search for the difference content between documents in the prior art.

[0045] Optionally, the acquisition of complexity evaluation indicators in S2 includes the following steps S23-S25:

[0046] S23, performing a ratio operation on the total number of characters in the text content of the document to be searched within a preset time period and the number of characters in the corresponding initial document to obtain an average number of characters, and obtaining character complexity according to the average number of characters.

[0047] S24. Monitor the paragraph structure of the text content in real time during the difference recognition process, and obtain the paragraph structure complexity coefficient according to the paragraph structure and a preset reference paragraph structure.

[0048] S25. Obtain a complexity evaluation index according to the character complexity, the paragraph structure complexity coefficient, the time complexity coefficient of the text comparison algorithm and the obtained recognition efficiency evaluation value.

[0049] Among them, the average number of characters is used to reflect the difficulty of the text content in the difference recognition process, and the character complexity is used to reflect the dynamic complexity of the evaluated text content in the difference recognition process; the paragraph structure represents the structural logic between sentences and paragraphs in the document to be searched, and the paragraph structure complexity coefficient is used to reflect the complexity of the paragraph structure of the text content within a preset time period.

[0050] In this embodiment, the average number of characters is the result of summing and averaging the ratio of the total number of characters in the text content within a preset time period to the number of characters in the initial document, wherein the number of characters in the initial document is the number of characters in the initial text content before difference recognition; the character complexity is a known value in the text comparison algorithm for measuring the character complexity of the text content during difference recognition, usually the product of the average number of characters divided by 2 and the time complexity coefficient; the paragraph structure complexity coefficient is the ratio of the paragraph structure complexity to the reference paragraph structure complexity, wherein the reference paragraph structure complexity is represented by summing and averaging the historical paragraph structure complexities in a preset database, for example, the reference paragraph structure is assumed to be "the main sentence contains 3 subordinate clauses". Now, we have two text paragraphs P1 and P2 for comparison. The main sentence of the P1 text content paragraph contains 5 clauses, and the corresponding paragraph structure complexity of P1 is 5 / 3≈1.67 (that is, the ratio of the number of clauses of P1 to the reference clause). The paragraph structure of P2 is that the main sentence contains 2 clauses, and the corresponding paragraph structure complexity of P2 is 2 / 3≈0.67, that is, the complexity of the paragraph structure of the P1 text content within the preset time period is greater than the complexity of the paragraph structure of the P2 text content within the preset time period.

[0051] Specifically, the complexity evaluation index is calculated by the following formula:

[0052]

[0053] In the formula, FU r.t M1 represents the complexity evaluation index of the rth document to be searched in the tth preset time period,r.t M2 represents the paragraph structure complexity coefficient of the rth document to be searched within the tth preset time period. r.t N represents the character complexity of the rth document to be searched in the tth preset time period, r.t represents the average number of characters of the rth document to be searched in the tth preset time period, δ r.t Indicates the time complexity coefficient of the rth document to be searched in the tth preset time period.

[0054] In a feasible implementation, in actual application, assuming that document A and document B represent versions at two different time points respectively, through real-time monitoring, it is identified that there are 100 differences between document A and document B, including 50 insertions, 30 deletions and 20 modifications, that is, the complexity evaluation index is calculated as: complexity score = number of differences × weight (such as insertion is 1, deletion is 1.5, modification is 2), then the score = 50×1+30×1.5+20×2=155.

[0055] Specifically, the statistical table of changes in complexity evaluation indicators is shown in Table 1:

[0056] Table 1 Statistics of changes in complexity evaluation indicators

[0057]

[0058] It should be understood that the time complexity coefficient is an existing parameter in the text comparison algorithm, which is usually used to describe the execution efficiency and resource consumption of the text comparison algorithm under different input scales. It is usually between 0 and 1, indicating the complexity of the difference recognition process of the document to be searched within a preset time period. For example, when the number of characters in the text content increases, the corresponding complexity increases; the complexity evaluation index decreases with the increase of the recognition efficiency evaluation value, and increases with the increase of the character complexity and paragraph structure complexity coefficients.

[0059] It should be noted that character complexity also indirectly affects the value of the paragraph structure complexity coefficient. As character complexity increases, the execution time of the text processing algorithm will be extended and the performance will decrease, resulting in incorrect division of the paragraph structure (such as dividing the first clause of the last main sentence of the paragraph into the previous main sentence), which in turn affects the accurate acquisition of the paragraph structure complexity coefficient.

[0060] This embodiment uses a text processing algorithm to reduce erroneous divisions caused by performance degradation, and uses natural language processing technology (such as deep learning and machine learning) to more accurately identify paragraph structures and sentence components in the text, so as to better understand the semantic information of the text, thereby more accurately dividing paragraphs and sentences and reducing the situation where different main clauses or subordinate clauses are incorrectly divided into adjacent paragraphs or sentences. This helps to improve the accuracy of the paragraph structure complexity coefficient, improves the accuracy and reliability of complexity evaluation indicators, and further improves the accuracy of searching for difference content between documents, effectively solving the problem of insufficient search for difference content between documents in the prior art.

[0061] Optionally, determining the difference identification result according to the complexity evaluation index in S2 includes the following steps S26-S28:

[0062] S26. Determine whether the obtained complexity evaluation index is equal to the preset minimum complexity value. If the complexity evaluation index is equal to the preset minimum complexity value, it indicates that the complexity evaluation index obtained in the difference identification process meets the complexity evaluation requirements, and the difference content obtained in the difference identification process is determined as the difference identification result. Otherwise, execute S27.

[0063] S27. Count the total number of difference types and difference distribution in the difference contents obtained in the difference identification process, and generate a complexity assessment report.

[0064] S28. Evaluate the difference content in the complexity assessment report to obtain the difference impact assessment value, repair the difference content according to the difference impact assessment value, and re-obtain the complexity assessment index according to the repaired difference content. When the re-obtained complexity assessment index is equal to the preset minimum complexity value, determine the repaired difference content as the difference identification result; otherwise, use the repaired difference content as the difference content obtained in the difference identification process and re-execute S27.

[0065] The difference types include addition, deletion and modification of text content, modification does not include addition and deletion, and the difference distribution is used to reflect the position of the difference content in the text content.

[0066] In a feasible implementation, the preset minimum complexity value is represented by the sum and average of the minimum complexity values ​​of historical document contents within the historical time period in a preset database; the difference content is the inconsistent text content between the text content within the preset time period and the text content within the previous preset time period. This setting can capture the changes in the text content to be found more finely, so as to fully find the differences in the text content to be found within the preset time period; then use a text comparison tool (such as Git difference display) to compare the text content between T1 and T2, and classify the identified differences into additions, deletions and modifications. Modifications should be defined as complex changes in the text content that include both deletions and additions, and are not simply additions or deletions. The difference distribution is usually the distribution of the number of difference classifications (i.e., the number of additions, the number of deletions and the number of modifications) in the complexity assessment report. In the text, the number of additions refers to the number of texts that exist in T2 but not in T1. For example, there is a sentence in T1, "The weather is very good today.", and "Let's go for a walk in the park" is added after this sentence in T2; the number of deletions refers to the number of texts that exist in T1 but not in T2. ​​For example, there is a sentence in T1, "We went shopping yesterday and bought a lot of things.", and this sentence was completely deleted in T2; the number of modifications usually involves the replacement or rewriting of the original text content. For example, there is a sentence in T1, "Our team is working hard to achieve the goals of the new project.", and this sentence in T2 was modified to "Our team has successfully completed the first phase of the goals of the new project and is ready to enter the next phase." There is both a replacement of vocabulary ("working hard to achieve" becomes "has been successfully completed") and an increase in content ("and is ready to enter the next phase").

[0067] Record the specific location of each difference (addition, deletion, modification) in the document (such as paragraph number, line number or character position), and then calculate the density of the difference content in the text (that is, the ratio of the number of characters in the difference content to the total number of characters in the text content), as well as the density differences of different parts; the complexity assessment report includes an overview (difference identification method, time range and text source) and difference statistics (total number of additions, deletions and modifications). By recording the specific location of the difference content in the document within a preset time period (such as paragraph number, line number or character position), this embodiment can provide more accurate difference positioning information than the prior art, and this accuracy helps users quickly locate specific changes in the text; by measuring the density of the difference content in the text (that is, the ratio of the number of characters in the difference content to the total number of characters in the text content), and comparing the density differences of different parts, it is possible to provide users with quantitative indicators of the distribution of the difference content in the text, which helps users understand the key distribution areas of the difference content, thereby better evaluating the degree and importance of text changes, achieving accurate identification of text content differences, and further achieving more accurate acquisition of complexity assessment reports.

[0068] Optionally, the difference impact assessment value can be calculated using the following formula (2):

[0069]

[0070] Where r is the number of the document to be searched, r = 1, 2, ..., R, R is the total number of documents to be searched, t is the number of the preset time period, t = 1, 2, ..., T, T is the total number of preset time periods, e is a natural constant, YIN r.t represents the difference impact evaluation value of the text content in the rth document to be searched in the tth preset time period, α1 represents the weight factor of the difference content ratio, W r.t ΔW0 represents the number of characters of the difference in the text content of the rth document to be searched within the tth preset time period, r.t represents the number of characters in the text content of the rth document to be searched within the tth preset time period, α2 represents the structural complexity weight factor, P r.t represents the structural complexity of the text content in the rth document to be searched within the tth preset time period, α3 represents the semantic complexity weight factor, Q r.t represents the semantic complexity of the text content in the rth document to be searched within the tth preset time period, FU r.t It represents the complexity evaluation index of the text content in the rth document to be searched in the tth preset time period.

[0071] In a feasible implementation manner, the difference content ratio is the ratio of the number of characters in the difference content to the number of characters in the text content; the structural complexity is obtained by converting the text content in the document to be searched into a structured representation (such as an abstract syntax tree) through an HTML parser, counting the types of structural elements (such as paragraphs, list items, and titles) and summing up the corresponding nesting depths (such as clauses in paragraphs, lists in lists, and levels of titles), wherein the nesting depth includes paragraph nesting depth (the ratio of the number of clauses in a paragraph to the total number of clauses in the text content), list nesting depth (the ratio of the number of lists in a list to the total number of lists in the text content), and title nesting depth (the number of levels in a title). The ratio of the number of levels to the total number of levels in the text content); the sentence structure in the text content is analyzed by the syntactic analyzer in natural language processing technology, and the semantic complexity is obtained by combining the sentence length, syntactic structure complexity and the number of semantic roles with the corresponding weights. For example, sentence length: 9 words, syntactic structure complexity: simple sentence, no clause, number of semantic roles: agent (subject), patient (object), dependency complexity: the number and type of edges are calculated according to the dependency graph. Assuming that the sentence length, syntactic structure complexity, number of semantic roles and dependency complexity are assigned different weights (such as 0.2, 0.3, 0.2, 0.3), the semantic complexity score of the sentence is: Among them, 20 is the standard number of characters in a line of text content, and 3 is a complete sentence component (subject, predicate, object).

[0072] Specifically, the difference impact assessment value increases with the increase of the number of characters in the difference content, the structural complexity and the semantic complexity. It should be noted that the number of characters in the difference content also indirectly affects the values ​​of the result complexity and semantic complexity. For example, a character difference may only be a spelling error of a word, while a difference containing multiple lines of text may involve structural changes in multiple paragraphs, such as paragraph rearrangement, sentence addition and reduction, which will increase the structural complexity and semantic complexity.

[0073] This embodiment not only considers the number of characters of the difference content, but also combines the structural complexity and semantic complexity for comprehensive evaluation. This comprehensive evaluation method can more comprehensively and accurately reflect the overall impact of the difference on the text. Compared with the existing technology that only focuses on the number of characters or a single dimension, the evaluation results of this embodiment are more reliable. Secondly, this embodiment can identify different types of differences (such as spelling errors, paragraph rearrangements, sentence additions and deletions). This difference classification and processing capability helps to more accurately evaluate the impact of differences on the structural complexity and semantic complexity of the text. Moreover, as the number of characters in the difference content increases, this embodiment can dynamically adjust the evaluation strategy and consider more factors (such as paragraph structure changes, semantic coherence) to evaluate the impact of the differences. This dynamic adjustment capability makes the evaluation results more adaptable to complex and changeable text differences.

[0074] The difference content ratio weight factor is a weight factor corresponding to the difference content ratio in the preset database, which indicates the numerical value of the influence degree of the difference content ratio on the difference impact assessment value. When used, the weight factor corresponding to the difference content ratio can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the difference content ratio corresponding to the proportion of the number of characters in the text content to the total number of characters and the weight factor corresponding to the difference content ratio in the preset database form a mapping set, and the real-time difference content ratio is input into the mapping set to obtain the corresponding weight factor, wherein the mapping relationship can be one-to-one or many-to-one. In this example, the value range of the difference content ratio weight factor is [0,1].

[0075] The structural complexity weight factor is the weight factor corresponding to the structural complexity in the preset database, which represents the numerical value of the influence of the structural complexity on the difference impact assessment value. When used, the weight factor corresponding to the structural complexity can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the structural complexity corresponding to the difference content and the weight factor corresponding to the structural complexity in the preset database form a mapping set, and the real-time structural complexity is input into the mapping set to obtain the corresponding weight factor, wherein the mapping relationship can be one-to-one or many-to-one. In this example, the value range of the structural complexity weight factor is [0,1]; the sum of the semantic complexity weight factor, the difference content ratio weight factor, and the structural complexity weight factor is 1.

[0076] In the scenario where the document to be found is for users or customers, by considering the proportion of differential content, structural complexity and semantic complexity relative to the differential impact assessment value, the impact of the differential content on the overall content of the document can be more accurately assessed, which helps to improve user satisfaction and trust in the document content. When the differential impact assessment value increases, it indicates that the corresponding differential content needs to be processed as a priority, thereby improving the accuracy and reliability of obtaining the differential impact assessment value, and further improving the accuracy of differential content search between documents, effectively solving the problem of insufficient differential content search between documents in the prior art.

[0077] S3. According to the highlighted visualization results, adjust the parameters of the text comparison algorithm, re-identify the differences according to the adjusted text comparison algorithm, and obtain the document difference score. The document difference score is used to measure the degree of difference between the documents to be found.

[0078] Optionally, the highlighting includes a diff location and a diff view:

[0079] Difference content positioning is used to scan the document to be searched line by line and word by word and compare it with the previous document to be searched to determine the difference position of the text content.

[0080] The difference content view is used to present the difference locations in the form of a view and add corresponding difference content annotations.

[0081] The view is the interactive interface between the accessing user and the document to be found, and the difference content annotation is attached to the difference position in the form of text boxes and tooltips.

[0082] In a feasible implementation manner, the main purpose of difference content positioning is to analyze the document to be searched, that is, to compare it with the previous document to be searched by scanning line by line and word by word. This comparison process is intended to identify the difference positions of the text content between the two documents, including but not limited to added, deleted or modified text fragments, and mark the scanned difference positions, which will be used for subsequent difference content view display; the difference content view is an interface that presents the results of difference content positioning to the user in an intuitive and easy-to-understand manner. In this view, the difference position will be highlighted (whole line, whole word or specific character), and corresponding difference content annotations will be attached. These annotations can be displayed directly next to the difference position in the form of a text box, or in the form of a tooltip when the user hovers the mouse over the difference position, so that the user can quickly identify and understand the differences between the documents; highlighting enables users to quickly browse and understand the differences between the documents to be searched, which helps to improve the efficiency and accuracy of finding differences between documents, and improves the accuracy and reliability of document difference visualization. The identified differences are visualized by highlighting (different colors can be used to mark different types of differences, such as red for deletion, green for insertion, and yellow for modification) so that users can intuitively see the changes in the document.

[0083] Optionally, parameters of the text comparison algorithm include a similarity threshold and a matching window.

[0084] S3 adjusts the parameters of the text comparison algorithm based on the highlighted visualization results, including:

[0085] When dissimilar texts are marked as differences, the similarity threshold is increased, and when similar texts are not marked as differences, the similarity threshold is decreased.

[0086] Increase the size of the matching window when there are discontinuities in highlighting that prevent the text comparison algorithm from capturing enough context to identify consecutive differences.

[0087] In a feasible implementation, when the highlighting increases, since the similarity threshold (the result of summing and averaging the historical similarities in a preset database represents the similarity threshold) is set low (such as 0.3), dissimilar texts are also marked as differences. At this time, the similarity threshold should be increased (such as from 0.3 to 0.5) to reduce unnecessary highlights. Otherwise, the similarity threshold should be lowered. The value of the similarity threshold is between 0 and 1; the matching window affects the context range considered by the algorithm when searching for matching text. If the highlighting is discontinuous, the algorithm cannot capture enough context to identify continuous differences. At this time, the size of the matching window should be increased (such as from 0.3 to 0.5), otherwise it should be reduced (such as from 0.3 to 0.1).

[0088] Optionally, obtain the difference score of the document from S3, including:

[0089] The number of differences in the difference content is monitored in real time, and the text complexity is obtained according to the number of differences and the corresponding reference number. The text complexity includes the complexity of newly added text, the complexity of deleted text and the complexity of modified text.

[0090] The preset new text complexity weight factor, deleted text complexity weight factor and modified text complexity weight factor are obtained from the preset database, and the document difference score is obtained by combining the obtained text complexity and difference impact evaluation value with the following formula (3).

[0091]

[0092] Where r is the number of the document to be searched, r = 1, 2, ..., R, R is the total number of documents to be searched, t is the number of the preset time period, t = 1, 2, ..., T, T is the total number of preset time periods, h is the number of the difference content row vector, h = 1, 2, ..., H, H is the total number of difference content row vectors, e is a natural constant, WEN r.t Indicates the document difference score of the rth document to be searched within the tth preset time period, YIN r.t represents the difference impact evaluation value of the text content in the rth document to be searched within the tth preset time period, represents the complexity of the newly added text in the text content of the hth line of the difference content in the rth document to be searched within the tth preset time period, ε1 represents the weight factor of the newly added text complexity, represents the complexity of deleting text in the text content of the hth line of the difference content in the rth document to be searched within the tth preset time period, ε2 represents the weight factor of the complexity of deleting text, represents the complexity of the modified text of the text content of the hth line of the difference content in the rth document to be searched within the tth preset time period, and ε3 represents the weight factor of the modified text complexity.

[0093] Among them, the new text complexity is the ratio of the number of new texts in the text content of the difference content row vector within the preset time period to the number of reference texts, the deleted text complexity is the ratio of the number of deleted texts in the text content of the difference content row vector within the preset time period to the number of reference texts, and the modified text complexity is the ratio of the number of modified texts in the text content of the difference content row vector within the preset time period to the number of reference texts.

[0094] In a feasible implementation, the preset difference score threshold is the difference between the maximum and minimum difference scores between historical documents in the preset database. It should be understood that the newly added text complexity weight factor is the weight factor corresponding to the newly added text complexity in the preset database, which represents the numerical value of the influence of the newly added text complexity on the document difference score. When used, the weight factor corresponding to the newly added text complexity can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the text complexity corresponding to the newly added difference content and the weight factor corresponding to the newly added text complexity in the preset database form a mapping set, and the real-time newly added text complexity is input into the mapping set to obtain the corresponding weight factor, wherein the mapping relationship can be one-to-one or many-to-one. The value range of the newly added text complexity weight factor in this example is [0,1].

[0095] The deleted text complexity weight factor is the weight factor corresponding to the deleted text complexity in the preset database, which represents the numerical value of the influence of the deleted text complexity on the document difference score. When used, the weight factor corresponding to the deleted text complexity can be directly obtained from the preset database, and the corresponding relationship can be a pre-set mapping relationship. For example, the text complexity corresponding to the deleted difference content and the weight factor corresponding to the deleted text complexity in the preset database form a mapping set, and the real-time deleted text complexity is input into the mapping set to obtain the corresponding weight factor, wherein the mapping relationship can be one-to-one or many-to-one. In this example, the value range of the deleted text complexity weight factor is [0,1]; the sum of the modified text complexity weight factor, the newly added text complexity weight factor, and the deleted text complexity weight factor is 1.

[0096] For example, the difference recognition is performed again using the adjusted text comparison algorithm. Based on the recognition results, the document difference score is calculated to measure the overall difference between the two versions of the document. For example, after adjusting the algorithm parameters, the difference recognition is performed again, and 80 differences are identified between document A and document B (which is a decrease from the 100 differences in the initial recognition, indicating that false positives and false negatives have been improved). The document difference score is calculated as: difference score = number of differences / total document length × 100%. Assuming the total length of the document is 1,000 words, the difference score = 80 / 1,000 × 100% = 8%.

[0097] In a feasible implementation, the document difference score increases with the increase of the difference impact assessment value, the complexity of the added text, the complexity of the deleted text, and the complexity of the modified text, and the difference impact assessment value also affects the values ​​of the complexity of the added text, the complexity of the deleted text, and the complexity of the modified text. The complexity of the added, deleted, and modified text is evaluated based on the complexity of the text content, which usually includes the diversity of vocabulary, the length and complexity of sentences, and the logical structure of paragraphs. Generally speaking, when the difference impact assessment value increases, the text changes involved may be more critical, so the text content involved in these changes is also more complex, thereby pushing up the complexity of the text. By considering the impact of the difference impact assessment value on the complexity of the added text, the complexity of the deleted text, and the complexity of the modified text, it helps to improve the accuracy of the text complexity in evaluating the differences between documents.

[0098] The prior art often identifies differences between documents based only on direct comparison of text content (such as word-by-word comparison). This method ignores the semantic complexity and impact behind the differences. The embodiments of the present invention introduce multiple dimensions such as difference impact evaluation value, new text complexity, deleted text complexity and modified text complexity. It not only considers the superficial changes in the text content of the document to be searched, but also deeply evaluates the impact of these changes on the overall content and structure of the document, thereby achieving accurate and comprehensive search of the difference content in the original document, thereby achieving comprehensiveness and accuracy in searching for difference content between documents. This complexity-based evaluation method ensures that difference identification is no longer limited to superficial text differences, but can provide insight into deeper semantic and structural changes, thereby greatly improving the accuracy of difference identification and effectively solving the problem of insufficient search for difference content between documents in the prior art.

[0099] After obtaining the document difference score, the method may further include the following steps:

[0100] Whether to execute the change assessment and feedback process is determined based on the document difference score. If not, the obtained document difference score is recorded and tracked. Otherwise, it indicates that the document difference score is greater than the preset difference score threshold and the difference type is re-acquired until the obtained document difference score is within the preset difference score threshold.

[0101] The embodiment of the present invention reflects the difficulty and number of difference identification through complexity evaluation indicators, and highlighting is an intuitive reflection of these differences in the document. By observing the visualization results of the highlighted display, the accuracy and completeness of the difference identification can be understood, thereby providing a basis for adjusting the algorithm parameters. The adjusted algorithm parameters will directly affect the results of the difference identification, thereby affecting the calculation of the document difference score. By continuously optimizing the algorithm parameters, the accuracy and completeness of the difference identification can be improved, thereby reducing the document difference score (when the number of differences is reduced) or more accurately reflecting the actual degree of difference between documents, thereby improving the accuracy of searching for difference content between documents.

[0102] Figure 3 is a block diagram of a device for finding differences between documents based on big data according to an exemplary embodiment, and the device is used to implement a method for finding differences between documents based on big data. Figure 3 The device includes a preprocessing module 310, a determination module 320 and an acquisition module 330. Among them:

[0103] A preprocessing module is used to obtain the document to be searched after format parsing, and preprocess the text content in the document to be searched;

[0104] A determination module is used to identify differences in the preprocessed text content based on a text comparison algorithm, obtain a complexity evaluation index, determine a difference identification result according to the complexity evaluation index, and visualize the difference identification result by highlighting;

[0105] The acquisition module is used to adjust the parameters of the text comparison algorithm according to the highlighted visualization results, re-identify the differences according to the adjusted text comparison algorithm, and obtain the document difference score; the document difference score is used to measure the degree of difference between the documents to be found.

[0106] In the embodiment of the present invention, during the writing and revision process of academic papers, documents often go through multiple versions of iterations. The embodiment of the present invention can accurately and quickly locate each changed content, including the addition, deletion and modification of text, by automatically comparing different versions of documents (such as drafts, first drafts, and final drafts). This not only helps the author to review and track the modification history, but also makes it easy for reviewers, editors, etc. to quickly understand the differences between versions, thereby improving the efficiency of reviewing and editing.

[0107] In real-time editing or review scenarios, such as online collaboration platforms or real-time document editors, embodiments of the present invention can instantly present the results of the difference identification module to the user in the form of color markings, such as red for deletion and green for addition. This instant feedback mechanism greatly enhances the user's editing and review experience, allowing them to quickly locate the differences and make corresponding adjustments or decisions.

[0108] In document editing, translation and other work, the document difference score can be used as an important indicator of quality control. By calculating the degree of difference between different versions of documents (such as the number of different words and sentences) and combining them with specific business needs (such as translation accuracy requirements and editing modification limits), the accuracy and efficiency of the work can be evaluated. When the difference score exceeds the preset range, the review process is automatically triggered to ensure that the work results meet the quality requirements.

[0109] In summary, it not only improves the efficiency and accuracy of difference identification, but also provides users with a more convenient and intuitive document processing experience, thereby achieving a more comprehensive search for differences between documents.

[0110] Figure 4 is a schematic diagram of a structure of a device for searching differences between documents based on big data provided by an embodiment of the present invention. Figure 4 As shown, the device for finding differences between documents based on big data may include the above Figure 3 The device for searching for differences between documents based on big data is shown. Optionally, the device 410 for searching for differences between documents based on big data may include a first processor 2001 .

[0111] Optionally, the device 410 for searching differences between documents based on big data may further include a memory 2002 and a transceiver 2003 .

[0112] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0113] Combine the following Figure 4 The components of the device 410 for finding differences between documents based on big data are specifically introduced as follows:

[0114] The first processor 2001 is the control center of the device 410 for searching differences between documents based on big data, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0115] Optionally, the first processor 2001 can perform various functions of the device 410 for finding differences between documents based on big data by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0116] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0117] In a specific implementation, as an embodiment, the device 410 for searching differences between documents based on big data may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0118] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0119] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0120] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0121] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0122] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0123] It should be noted that Figure 4 The structure of the device 410 for finding differences between documents based on big data shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0124] In addition, the technical effects of the device 410 for finding differences between documents based on big data can refer to the technical effects of the method for finding differences between documents based on big data described in the above method embodiment, and will not be repeated here.

[0125] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0126] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0127] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0128] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0129] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0130] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0131] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0137] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for finding differences between documents based on big data, characterized in that: The method comprises: S1, obtaining a document to be searched after format analysis, and preprocessing the text content in the document to be searched; S2. Based on the text comparison algorithm, the preprocessed text content is identified for differences, and a complexity evaluation index is obtained. The difference identification result is determined according to the complexity evaluation index, and the difference identification result is visualized by highlighting; S3. According to the highlighted visualization results, adjust the parameters of the text comparison algorithm, re-perform difference identification according to the adjusted text comparison algorithm, and obtain a document difference score; the document difference score is used to measure the degree of difference between the documents to be found.

2. The method for finding differences between documents based on big data according to claim 1, characterized in that: The text comparison algorithm of S2 performs difference recognition on the preprocessed text content, including: S21, converting the text content after removing the stop words into a numerical vector and performing a morphological normalization operation to obtain a common subsequence in the document to be searched in the current preset time period and the previous preset time period, wherein the morphological normalization operation includes temporal normalization and singular-plural normalization; S22. Obtain the edit distance in the common subsequence using a dynamic programming algorithm, and record the differences of corresponding positions in the common subsequence in real time according to the edit distance.

3. The method for finding differences between documents based on big data according to claim 1, characterized in that: After the text comparison algorithm is used to identify differences in the preprocessed text content, the method further includes: Obtain a recognition efficiency evaluation value according to the following formula (1), wherein the recognition efficiency evaluation value is used to measure the difference recognition efficiency of the text content in the document to be searched within the current preset time period; Where r is the number of the document to be searched, r = 1, 2, ..., R, R is the total number of documents to be searched, t is the number of the preset time period, t = 1, 2, ..., T, T is the total number of preset time periods, e is a natural constant, YI r.t A1 represents the recognition efficiency evaluation value of the rth document to be searched in the tth preset time period, r.t A2 represents the actual number of preprocessed characters of the rth document to be searched in the tth preset time period. r.t represents the actual number of difference recognition characters of the rth document to be searched in the tth preset time period, ΔA1 r.t represents the number of reference preprocessed characters of the rth document to be searched within the tth preset time period, ΔA2 r.t T1 represents the number of reference difference recognition characters of the rth document to be searched within the tth preset time period, r.t represents the actual preprocessing time of the rth document to be searched in the tth preset time period, T2 r.t Indicates the actual difference recognition time of the rth document to be searched within the tth preset time period, ΔT1 r.t represents the reference preprocessing time of the rth document to be searched in the tth preset time period, ΔT2 r.t Indicates the reference difference recognition time of the rth document to be searched within the tth preset time period.

4. The method for finding differences between documents based on big data according to claim 3, characterized in that: The acquisition complexity evaluation indicators of S2 include: S23, performing a ratio operation on the total number of characters in the text content of the to-be-searched document within a preset time period and the number of characters in the corresponding initial document to obtain an average number of characters, and obtaining character complexity according to the average number of characters; S24, monitoring the paragraph structure of the text content in real time during the difference identification process, and obtaining a paragraph structure complexity coefficient according to the paragraph structure and a preset reference paragraph structure; S25. Obtain a complexity evaluation index according to the character complexity, the paragraph structure complexity coefficient, the time complexity coefficient of the text comparison algorithm and the obtained recognition efficiency evaluation value.

5. The method for finding differences between documents based on big data according to claim 4 is characterized in that: The step S2 determines the difference recognition result according to the complexity evaluation index, including: S26, judging whether the obtained complexity evaluation index is equal to the preset minimum complexity value, if the complexity evaluation index is equal to the preset minimum complexity value, it indicates that the complexity evaluation index obtained in the difference identification process meets the complexity evaluation requirement, and the difference content obtained in the difference identification process is determined as the difference identification result, otherwise, executing S27; S27, counting the total number of difference types and difference distribution in the difference contents obtained in the difference identification process, and generating a complexity assessment report; S28. Evaluate the difference content in the complexity assessment report to obtain the difference impact assessment value, repair the difference content according to the difference impact assessment value, and re-obtain the complexity assessment index according to the repaired difference content. When the re-obtained complexity assessment index is equal to the preset minimum complexity value, determine the repaired difference content as the difference identification result; otherwise, use the repaired difference content as the difference content obtained in the difference identification process and re-execute S27.

6. The method for finding differences between documents based on big data according to claim 5, characterized in that: The difference impact assessment value is calculated by the following formula (2): Where r is the number of the document to be searched, r = 1, 2, ..., R, R is the total number of documents to be searched, t is the number of the preset time period, t = 1, 2, ..., T, T is the total number of preset time periods, e is a natural constant, YIN r.t represents the difference impact evaluation value of the text content in the rth document to be searched in the tth preset time period, α1 represents the weight factor of the difference content ratio, W r.t ΔW0 represents the number of characters of the difference in the text content of the rth document to be searched within the tth preset time period, r.t represents the number of characters in the text content of the rth document to be searched within the tth preset time period, α2 represents the structural complexity weight factor, P r.t represents the structural complexity of the text content in the rth document to be searched within the tth preset time period, α3 represents the semantic complexity weight factor, Q r.t represents the semantic complexity of the text content in the rth document to be searched within the tth preset time period, FU r.t It represents the complexity evaluation index of the text content in the rth document to be searched in the tth preset time period.

7. The method for finding differences between documents based on big data according to claim 6, characterized in that: The highlighting includes a difference content location and a difference content view; The difference content positioning is used to scan the document to be searched line by line and word by word and compare it with the previous document to be searched to determine the difference position of the text content; The difference content view is used to present the difference position in a view form and add corresponding difference content annotations.

8. The method for finding differences between documents based on big data according to claim 7, characterized in that: The parameters of the text comparison algorithm include similarity threshold, matching window and word segmentation granularity; S3 adjusts the parameters of the text comparison algorithm based on the highlighted visualization results, including: When dissimilar text is marked as a difference, the similarity threshold is increased, and when similar text is not marked as a difference, the similarity threshold is decreased; Increase the size of the matching window when there are discontinuities in highlighting that prevent the text comparison algorithm from capturing enough context to identify consecutive differences; When professional terms are split, increase the word segmentation granularity.

9. The method for finding differences between documents based on big data according to claim 8, characterized in that: S3 retrieved document difference scores, including: Monitor the number of differences in the difference content in real time, and obtain text complexity according to the number of differences and the corresponding reference number, wherein the text complexity includes the complexity of newly added text, the complexity of deleted text, and the complexity of modified text; Obtaining the preset new text complexity weight factor, deleted text complexity weight factor and modified text complexity weight factor from the preset database, and combining the obtained text complexity and difference impact assessment value with the following formula (3) to obtain the document difference score; Where r is the number of the document to be searched, r = 1, 2, ..., R, R is the total number of documents to be searched, t is the number of the preset time period, t = 1, 2, ..., T, T is the total number of preset time periods, h is the number of the difference content row vector, h = 1, 2, ..., H, H is the total number of difference content row vectors, e is a natural constant, WEN r.t Indicates the document difference score of the rth document to be searched within the tth preset time period, YIN r.t represents the difference impact evaluation value of the text content in the rth document to be searched within the tth preset time period, represents the complexity of the newly added text in the text content of the hth line of the difference content in the rth document to be searched within the tth preset time period, ε1 represents the weight factor of the newly added text complexity, represents the complexity of deleting text in the text content of the hth line of the difference content in the rth document to be searched within the tth preset time period, ε2 represents the weight factor of the complexity of deleting text, represents the complexity of the modified text of the text content of the hth line of the difference content in the rth document to be searched within the tth preset time period, and ε3 represents the weight factor of the modified text complexity; After obtaining the document difference score, the method further includes: Whether to execute the change assessment and feedback process is determined based on the document difference score. If not, the obtained document difference score is recorded and tracked. Otherwise, it indicates that the document difference score is greater than the preset difference score threshold, and the difference type is re-obtained until the obtained document difference score is within the preset difference score threshold.

10. A device for searching for differences between documents based on big data, the device for searching for differences between documents based on big data being used to implement the method for searching for differences between documents based on big data as claimed in any one of claims 1 to 9, characterized in that: The device comprises: A preprocessing module is used to obtain the document to be searched after format parsing, and preprocess the text content in the document to be searched; A determination module is used to identify differences in the preprocessed text content based on a text comparison algorithm, obtain a complexity evaluation index, determine a difference identification result according to the complexity evaluation index, and visualize the difference identification result by highlighting; The acquisition module is used to adjust the parameters of the text comparison algorithm according to the highlighted visualization results, re-identify the differences according to the adjusted text comparison algorithm, and obtain the document difference score; the document difference score is used to measure the degree of difference between the documents to be found.

Images