Document difference comparison method and device and storage medium

Abstract

The invention discloses a document difference comparison method and device and a storage medium, and relates to the technical field of computers.The method comprises the steps that difference comparison subtasks corresponding to documents to be compared are obtained, and the difference comparison subtasks are added into a multi-thread task queue according to the priorities of the difference comparison subtasks; extracting at least two target sub-tasks from the multi-thread task queue, and synchronously performing difference comparison on the target sub-tasks according to a preset large model in a processing process corresponding to the extracted target sub-tasks one by one to obtain a first difference result; and after all the target sub-tasks are executed, obtaining the first difference results, and integrating the first difference results to obtain a difference report. According to the method, the difference comparison analysis can be performed on the long document, and the comparison efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a document difference comparison method, device and storage medium. Background Art

[0002] Large-scale pre-trained language models (such as GPT-3, BERT, etc.) are widely used in the field of document difference comparison, such as: common applications include legal document review, academic paper comparison, code review, and corporate report analysis. Large-scale pre-trained language models can understand the semantics of text and provide accurate and detailed difference analysis when comparing two documents.

[0003] However, large models have strict restrictions on input length, usually limited to between 512 and 4096 tokens, which makes it impossible to directly process long documents. Currently, long documents are usually processed in blocks, and the difference analysis and identification of the documents corresponding to each block are performed one by one. After the difference comparison of the documents corresponding to a block is completed, other blocks are processed. The block processing and block-by-block comparison of long documents increases the computing overhead and time consumption, and reduces the processing efficiency.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a document difference comparison method, device and storage medium, aiming to solve the technical problem of how to improve the efficiency of difference analysis on long texts.

[0006] To achieve the above purpose, the present application proposes a document difference comparison method, which includes:

[0007] Obtaining the difference comparison subtasks corresponding to the documents to be compared, and adding each of the difference comparison subtasks to a multi-threaded task queue according to the priority of each of the difference comparison subtasks;

[0008] Extracting at least two target subtasks from the multi-threaded task queue, and in a processing process corresponding to the extracted target subtasks one by one, synchronously performing a difference comparison on the target subtasks according to a preset large model to obtain a first difference result;

[0009] After all the target subtasks are executed, each of the first difference results is obtained, and the first difference results are integrated to obtain a difference report.

[0010] In one embodiment, before the step of obtaining the difference comparison subtasks corresponding to the to-be-processed documents and adding each of the difference comparison subtasks to the multi-threaded task queue according to the priority of each of the difference comparison subtasks, the step further includes:

[0011] Acquire the document to be compared, and perform chapter matching on the document to be compared according to semantic similarity to obtain paired chapters;

[0012] Determine a slicing point in the paired chapters, and obtain the number of characters of the segment corresponding to the slicing point;

[0013] If the number of characters is less than or equal to a preset first character number threshold, determining the slicing point as a target slicing point;

[0014] According to the target slicing points, each of the paired chapters is segmented to obtain a target segment, and the difference comparison subtask is created for each of the target segments.

[0015] In one embodiment, the step of obtaining the document to be compared and performing chapter matching on the document to be compared according to semantic similarity to obtain paired chapters includes:

[0016] Calculating the semantic similarity of each chapter of the document to be compared according to a semantic analysis algorithm to obtain a similarity value;

[0017] Determining whether the similarity value is greater than a preset similarity threshold;

[0018] If it is greater, the corresponding chapters are paired to obtain paired chapters.

[0019] In one embodiment, the step of extracting at least two target subtasks from the multi-threaded task queue, and synchronously performing a difference comparison on the target subtasks according to a preset large model in a processing process corresponding to the extracted target subtasks, and obtaining a first difference result includes:

[0020] Constructing a prompt engineering template, wherein the prompt engineering template includes: comparison instructions and output format;

[0021] Input the target subtask and the prompt engineering template into the preset large model to compare and analyze the differences;

[0022] Based on the preset large model, the first difference result that conforms to the construction prompt engineering template is generated.

[0023] In one embodiment, the method further comprises:

[0024] During the difference comparison process, detecting the number of characters in the target segment corresponding to the target subtask;

[0025] When the number of characters in the target segment is greater than or equal to a preset second character number threshold, execution of the target subtask is terminated.

[0026] In one embodiment, after the step of terminating the execution of the target subtask when the number of characters of the target segment is greater than or equal to a preset second character number threshold, the step further includes:

[0027] Obtaining unprocessed document segments exceeding the second character count threshold;

[0028] An unprocessed subtask is created for the unprocessed document fragment, and the large model performs a difference comparison on the unprocessed subtask.

[0029] In one embodiment, before the step of obtaining the difference comparison subtasks corresponding to the documents to be compared and adding each of the difference comparison subtasks to the multi-threaded task queue according to the priority of each of the difference comparison subtasks, the following steps are included:

[0030] Obtaining the number of target subtasks being executed, and determining whether the number is less than a preset execution number threshold;

[0031] If it is less than, the difference comparison subtask corresponding to the to-be-compared document is obtained, and each difference comparison subtask is added to the multi-threaded task queue according to the priority of each difference comparison subtask.

[0032] In one embodiment, the step of extracting at least two target subtasks from the multi-threaded task queue and processing the extracted target subtasks in a one-to-one correspondence with each other comprises:

[0033] Extracting the target subtasks in order of the number of characters in the documents corresponding to the difference comparison subtasks from small to large;

[0034] Allocating the target subtasks to different processing processes in sequence according to the order in which the target subtasks are extracted;

[0035] After being assigned to the last processing process, the target subtask returns to the first processing process for assignment until all the target subtasks are assigned.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a document difference comparison device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document difference comparison method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document difference comparison method described above are implemented.

[0038] The present application provides a document difference comparison method, in which the number of characters of the comparison documents contained in each document difference comparison subtask meets the input length restriction of the large model. By adding the document difference comparison subtask to the multi-threaded task queue, the multi-threaded synchronous execution of the document difference comparison subtask can reduce the computing time and resource consumption, improve the document processing speed, and shorten the user waiting time. By setting the priority of the difference comparison subtask, the number of tasks processed at the same time can be limited to avoid system crashes or performance degradation caused by task overload. When system resources are tight, high-priority tasks can be processed first, and low-priority tasks can be temporarily shelved to maintain the stable operation of the system. By using a large model to perform document difference comparison and obtain difference results, the large model can usually capture more details and complex relationships, and can improve the accuracy and comprehensiveness of the comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A flowchart of the first embodiment of the method for comparing document differences in this application;

[0042] Figure 2 A detailed flow chart of the first embodiment of the document difference comparison method of this application;

[0043] Figure 3 Another detailed flow chart provided for Example 1 of the document difference comparison method of this application;

[0044] Figure 4 A flowchart of the second embodiment of the document difference comparison method of this application is provided;

[0045] Figure 5 A detailed flow chart of Example 2 of the document difference comparison method of this application;

[0046] Figure 6 A flowchart of the third embodiment of the document difference comparison method of this application is provided;

[0047] Figure 7 A flowchart of the fourth embodiment of the document difference comparison method of this application is provided;

[0048] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the document difference comparison method in the embodiment of the present application. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] The main solution of the embodiment of the present application is: obtaining the difference comparison subtask corresponding to the document to be compared, and adding each of the difference comparison subtasks to the multi-threaded task queue according to the priority of each of the difference comparison subtasks; extracting at least two target subtasks from the multi-threaded task queue, and in the processing process corresponding to the extracted target subtasks one by one, synchronously performing difference comparison on the target subtasks according to the preset large model to obtain a first difference result; after all the target subtasks are executed, obtaining each of the first difference results, and integrating the first difference results to obtain a difference report.

[0052] Large-scale pre-trained language models (such as GPT-3, BERT, etc.) are widely used in the field of document difference comparison, such as: common applications include legal document review, academic paper comparison, code review, and corporate report analysis. Large-scale pre-trained language models can understand the semantics of text and provide accurate and detailed difference analysis when comparing two documents.

[0053] However, large models have strict restrictions on input length, usually limited to between 512 and 4096 tokens, which makes it impossible to directly process long documents. Currently, long documents are usually processed in blocks, and then differential analysis is performed based on large-scale pre-trained language models. Block processing and block-by-block comparison of long documents increase computing overhead and time consumption, and reduce processing efficiency.

[0054] The present application provides a document difference comparison method, in which the number of characters of the comparison documents contained in each document difference comparison subtask meets the input length restriction of the large model. By adding the document difference comparison subtask to the multi-threaded task queue, the multi-threaded synchronous execution of the document difference comparison subtask can reduce the computing time and resource consumption, improve the document processing speed, and shorten the user waiting time. By setting the priority of the difference comparison subtask, the number of tasks processed at the same time can be limited to avoid system crashes or performance degradation caused by task overload. When system resources are tight, high-priority tasks can be processed first, and low-priority tasks can be temporarily shelved to maintain the stable operation of the system. By using a large model to perform document difference comparison and obtain difference results, the large model can usually capture more details and complex relationships, and can improve the accuracy and comprehensiveness of the comparison.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a device, etc. that can realize the above functions. The following takes a document difference comparison device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a document difference comparison method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the document difference comparison method of this application.

[0057] In this embodiment, the document difference comparison method includes steps S100 to S300:

[0058] Step S100: obtaining the difference comparison subtasks corresponding to the documents to be compared, and adding each of the difference comparison subtasks to a multi-threaded task queue according to the priority of each of the difference comparison subtasks.

[0059] It should be noted that a multithreaded queue refers to a queue data structure used in a multithreaded programming environment. A queue is a first-in-first-out (FIFO) data structure in which elements are arranged in the order in which they enter the queue, and the element that enters the queue first is taken out first.

[0060] It should be noted that the document difference comparison subtask refers to splitting the overall document difference comparison task into multiple smaller, independently executable subtasks, each of which involves comparing one or more specific document parts.

[0061] In this embodiment, a multi-threaded task queue is created in advance, and after all document difference comparison subtasks are obtained, a priority is set for each subtask, and the priority setting can be determined according to factors such as the importance of the subtask or the estimated processing time. Each subtask is added to the multi-threaded task queue in the preset priority order.

[0062] Exemplarily, when the document difference comparison subtasks are prioritized according to the estimated processing time, for subtasks with a longer estimated processing time and requiring more computing resources (such as document comparisons containing large amounts of data or complex formats), their priorities can be set to a lower level so that they can be processed when resources are idle. For subtasks with a shorter estimated processing time and easy to complete (such as simple text comparisons, format checks, etc.), their priorities can be set to a higher level to quickly complete these tasks and free up resources. When the document difference comparison subtasks are prioritized according to the importance of the subtasks, if a subtask involves a critical document (such as a contract, legal document, important report, etc.), its priority should be set to the highest level. For documents that are frequently modified, their difference comparison subtasks should also have a higher priority to ensure that the content of the latest version is compared and analyzed in a timely manner.

[0063] Step S200, extracting at least two target subtasks from the multi-threaded task queue, and in a processing process corresponding to the extracted target subtasks one by one, synchronously performing a difference comparison on the target subtasks according to a preset large model to obtain a first difference result.

[0064] In this embodiment, a task start condition is preset, and the task start condition may include the number of concurrent tasks and the computing resources reaching a certain state. When the preset task start condition is met, one or more target subtasks are extracted from the task queue in order of priority, and these target subtasks are assigned to corresponding threads for asynchronous execution. Each thread is responsible for executing the subtask assigned to it, that is, comparing a pair of chapters and generating a summary of the differences.

[0065] In this embodiment, before executing the target subtask, check whether the preset task start condition is met. The task start condition is that the task queue is not empty. Only when there are pending subtasks in the task queue will multi-threaded execution be considered. The task start condition can also be sufficient computing resources. The number of multi-threads that can be supported by resources such as CPU, memory, and disk I / O can be set. If the current computing resources are less than the preset threshold, the task execution is started. The task start condition can also be the number of concurrent tasks. If the current number of concurrent tasks is less than the upper limit of the number, a new thread is started to execute the task. Through reasonable task start conditions, document difference comparison tasks can be efficiently processed to avoid task overload or resource exhaustion.

[0066] For example, when the CPU usage is less than 80% and the memory usage is less than 70%, it is considered that there are sufficient computing resources to start a new thread. The upper limit of concurrent tasks is 10. If the current number of concurrent tasks is less than 10, a new thread is started to execute the task. If the current number of concurrent tasks has reached or exceeded 10, wait for the existing task to complete.

[0067] In this embodiment, the thread receives the target subtask assigned to it from the task queue. These subtasks usually include a pair of document sections that need to be compared, as well as the parameters and configurations required for the comparison. The preprocessed text is input into the preset large model, which will perform in-depth analysis and understanding of the text, and then output the results of the difference comparison. These results may include information such as the location, type, and degree of the difference points. Based on the output of the model, the thread generates a first difference result. This result may be a list containing detailed information about the difference points, a copy of the document with the difference points marked, or a difference report. The large model can more accurately understand the meaning and context of the text, thereby more accurately identifying the differences between documents.

[0068] Step S300, after all the target subtasks are executed, each of the first difference results is obtained, and the first difference results are integrated to obtain a difference report.

[0069] In this embodiment, after all subtasks are completed, the first difference result of each subtask is obtained, and all the first difference results are summarized to form a complete first difference result list. Then, the summarized first difference result list is used as input, and the pre-trained large language model is input to summarize and refine the differences of the entire document, and a full-text difference report is generated. The format of the difference report can be Markdown format.

[0070] In this embodiment, when integrating the first difference results, the merged difference points can be sorted according to the importance, frequency of occurrence or other criteria of the difference points. The difference points can also be classified according to the document structure (such as chapters). The sorted and classified difference points are integrated into a complete difference point list.

[0071] Optionally, create a task status table to track the execution status of each task. When a thread takes a task from the task queue and starts executing it, it locks the task status table, updates the task status to "in progress", and unlocks it after the task status update is completed. After the task is executed, the thread locks the task status table again and updates the task status to "completed". During the task execution, if an error is encountered, the task status is updated to "failed". Check the task status table regularly, traverse the task status table and count the number of tasks in the "unfinished" state. Once all tasks have been completed, perform the steps of summarizing the results and outputting reports.

[0072] In the present embodiment, during the task execution process, a try-except structure is used to capture the exception that may be thrown. After catching the exception, the error information (including the exception type, message, stack trace, etc.) is recorded in a log file. Exemplarily, a log library (such as Python's logging module) can be used to record errors. It is determined whether to retry the failed task according to the exception type, and the exception type includes but is not limited to: file read errors caused by file path errors, file corruption or file permission problems, content parsing errors caused by incorrect file format, encoding problems or content not meeting expectations, network connection errors, and insufficient memory errors. If you decide to retry, put the task back into the task queue. And set an appropriate number of retries or delays. If the task still fails after reaching the maximum number of retries, it is marked as a "failed" state and removed from the task queue. And the information of the failed task is notified to the relevant personnel or system.

[0073] For example, for file read errors, the number of retries is 3, with a delay of 5 seconds each time. For content parsing errors, an error is reported directly without retrying. For network connection errors, the number of retries is 5, with an initial delay of 2 seconds, and the delay time doubles after each retry. For insufficient memory errors, the number of retries is 2.

[0074] Please refer to Figure 2 In a feasible implementation manner, step S200 may include steps S210 to S230:

[0075] Step S210, constructing a prompt engineering template, wherein the prompt engineering template includes: comparison instructions and output format;

[0076] Step S220, inputting the target subtask and the prompt engineering template into the preset macro model to perform difference comparison and analysis;

[0077] Step S230: generating the first difference result that complies with the construction prompt engineering template based on the preset large model.

[0078] It should be noted that the prompt engineering template is a tool used in the field of artificial intelligence and natural language processing, which aims to optimize the generation output of large language models (such as the GPT series, BERT series, etc.) by providing structured input guidance. The prompt engineering template usually contains a series of carefully designed prompts or instructions that can guide the model to generate text that meets specific requirements or formats.

[0079] In this embodiment, the constructed prompt engineering template specifies the comparison instructions and output format. The comparison instructions are used to guide the large model to perform a difference comparison on two or more objects. The dimensions, scope, and key points of the comparison are specified. The output format is used to standardize the content and form of the large model output to ensure that the output information is clear and easy to understand. The specific document content fragment corresponding to the target subtask is combined with the comparison instructions in the prompt engineering template to form a complete input instruction. The input instruction is input into the preset large model, and the large model performs a difference comparison and analysis on the target subtask according to the input instruction. The large model generates a first difference comparison result according to the requirements of the comparison instructions and the output format. The first difference comparison result contains the difference points of all dimensions specified in the comparison instruction (such as content, structure, language style, etc.).

[0080] Optionally, the generated first difference result is verified to ensure that the result meets the requirements of the output format. When it is found that the result does not meet the requirements or there are errors, the comparison instructions or output format can be adjusted and re-entered into the large model for processing. Exemplarily, an automated verification tool, such as a JSON Schema validator, a regular expression matcher, etc., is selected for format verification to check whether the output result meets the preset format requirements. The content verification script is executed to evaluate the difference results item by item. If the content is found to be inaccurate, incomplete, or there are problems such as false positives and false negatives, the error information is recorded. For errors found in format verification and content verification, the cause of the error is analyzed. If it is caused by improper setting of the comparison instruction or output format, adjust the corresponding instruction or format, and re-enter the large model for processing. If it is caused by the performance or error of the large model itself, record the error case and consider tuning or repairing the model. Finally, the verification results and error handling are fed back to the relevant personnel so that they can understand the task progress and existing problems. By verifying the generated first difference result, the generated difference result can be comprehensively and carefully verified to ensure that the result meets the requirements of the output format and accurately reflects the differences between the target subtasks.

[0081] Exemplarily, the comparison instructions in the prompt engineering template may include the following contents:

[0082] Please compare the following two documents and find the differences between them based on the user's question, including only the following aspects:

[0083] 1. **Content differences**: including additions or deletions of text content, such as whether certain key information has been added, deleted, or adjusted.

[0084] 2. **Data and values**: including changes in values, such as whether the data, values, dates, etc. involved have changed; statistical information, such as statistical tables, increases or decreases in data, changes in calculation methods, or differences in results.

[0085] Please ignore formatting differences, structural differences, language and grammatical differences, and only return up to 10 differences, and be as precise as possible.

[0086] For each difference, provide the following details:

[0087] -**Summary of Differences**: Briefly describe the differences found.

[0088] - **Original text snippet of Document 1**: Provide a snippet of no more than 200 words from the relevant part of Document 1.

[0089] - **Original text snippet of Document 2**: Provide a snippet of no more than 200 words from the relevant part of Document 2.

[0090] - **The location of Document 1** (such as page number, chapter, clause number, etc.).

[0091] - **The location of Document 2** (such as page number, chapter, clause number, etc.).

[0092] Tip: The output format requirements in the project template may include the following:

[0093] ###Output format (using Markdown format)

[0094] ˋˋˋmarkdown

[0095] List of differences:

[0096] 1. **Summary of Differences**: Describe the differences found in the comparison.

[0097] -**Document 1 original text segment**: ˋ“Original text segment in Document 1”ˋ

[0098] -**Document 2 original text segment**: ˋ“Original text segment in Document 2”ˋ

[0099] - **Document 1 Location**: Page X, Chapter X, Article X.

[0100] -**Document 2 Location**: Page X, Chapter X, Article X.

[0101] ```

[0102] ###Notes

[0103] - The original document may not be altered.

[0104] - Do not fabricate differences.

[0105] The text of Document 1 is as follows:

[0106] ```Markdown

[0107] {context1}

[0108] ```

[0109] The text of Document 2 is as follows:

[0110] ```Markdown

[0111] {context2}

[0112] ```

[0113] User Question:

[0114] ```Markdown

[0115] {question}

[0116] ```

[0117] In this embodiment, by constructing a prompt engineering template that meets the requirements, and combining the target subtask with the template and inputting it into a preset large model for difference comparison and analysis, a first difference result that meets the construction prompt engineering template can be generated, providing accurate and clear comparison information.

[0118] Please refer to Figure 3 In a feasible implementation manner, before step S100, steps C100 to C200 may be further included:

[0119] Step C100, obtaining the number of target subtasks being executed, and determining whether the number is less than a preset execution number threshold;

[0120] Step C200: If it is less than, then obtain the difference comparison subtask corresponding to the document to be compared, and add each difference comparison subtask to the multi-threaded task queue according to the priority of each difference comparison subtask.

[0121] In this embodiment, the number of target subtasks currently being executed is tracked in real time according to a counter or a status list. Whenever a new task is started, the counter is incremented; when the task is completed, the counter is decremented. A threshold value of the number of executions is preset, which represents the maximum number of tasks that can be processed at the same time. The number of tasks being executed is checked regularly and compared with the preset threshold. If the number of tasks currently being executed is less than the preset threshold, it is considered that there are sufficient resources to process new tasks. At this time, the new task is started and added to the task queue to wait for execution.

[0122] In this implementation, a token bucket algorithm can be used to control task execution. Create a token bucket to store a fixed number of tokens. Each token represents the authority of the system to process a task. The token bucket adds tokens to the bucket at a certain rate (such as generating one token per second). When a new target subtask needs to be executed, it first tries to get a token from the token bucket. If there is a token in the bucket, the task successfully obtains the token and starts execution immediately. At the same time, a token is removed from the bucket. If there is no token in the bucket, the task needs to wait until there is an available token in the bucket. The size of the token bucket and the token generation rate can be dynamically adjusted according to the actual load and performance of the system.

[0123] In the first embodiment of the present application, the number of characters of the comparison documents contained in each document difference comparison subtask meets the input length restriction of the large model. By adding the document difference comparison subtask to the multi-threaded task queue, the multi-threaded synchronous execution of the document difference comparison subtask can reduce the computing time and resource consumption, improve the document processing speed, and shorten the user waiting time. By setting the priority of the difference comparison subtask, the number of tasks processed at the same time can be limited to avoid system crashes or performance degradation caused by task overload. When system resources are tight, high-priority tasks can be processed first, and low-priority tasks can be temporarily shelved to maintain the stable operation of the system. By using the large model to perform document difference comparison and obtain difference results, the large model can usually capture more details and complex relationships, and can improve the accuracy and comprehensiveness of the comparison.

[0124] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , before step S100, steps A100 to A400 are also included:

[0125] Step A100, obtaining the document to be compared, and performing chapter matching on the document to be compared according to semantic similarity to obtain paired chapters;

[0126] Step A200, determining a slicing point in the paired chapter, and obtaining the number of characters of the segment corresponding to the slicing point;

[0127] Step A300, if the number of characters is less than or equal to a preset first character number threshold, determining the slicing point as a target slicing point;

[0128] Step A400, segmenting each of the paired chapters according to the target slicing points to obtain target segments, and creating the difference comparison subtask for each of the target segments.

[0129] It should be noted that, in this embodiment, the number of characters refers to the number of tkoen. Token is a special computer term, commonly used in data management and verification of network communications. It represents the identifier of a data element and can represent a specific data type, such as a number, a string, an array, etc. In the field of natural language processing (NLP), especially in large language models (such as ChatGPT), token is the basic unit of model input and can be understood as the smallest unit in the text. In English, it is usually a word or punctuation mark, and in Chinese, characters or words are usually used as tokens.

[0130] In this embodiment, the document to be compared is obtained from a specified source (such as a file system, a database, a cloud storage, etc.). Natural language processing (NLP) technology, such as text vectorization, semantic analysis, similarity calculation, etc., is used to evaluate the semantic similarity of the chapters of the document to be compared. According to the evaluation results, the semantically similar chapters are paired to form paired chapters. Within the chapter, possible slicing points are obtained, which are usually located at the end of the paragraph, at the topic transition, after the important argument or conclusion. Ensure that each slicing point can cut the chapter into a complete unit of meaning. Use a text editor or a special character counter to determine the number of characters of each potential slicing point, and select the target slicing point based on the character limit and semantic integrity. The target slicing point not only meets the character number requirement of the first character number threshold, but also ensures that the content after slicing is still complete and coherent when read independently. The segment after the chapter is segmented according to the target slicing point is called the target segment. For each target segment, a document difference comparison subtask is created. The subtask can contain the identification of the paired chapter segment, the source document information, the comparison requirements, etc. The subtask can be stored in the task queue, waiting for subsequent processing and execution.

[0131] In this implementation, the semantic understanding ability of the large model can be fully utilized even when the number of tokens is limited. Optimizing the document comparison effect under the limitation of short input tokens: Through innovative methods, the large model can process long documents more effectively, avoid the loss of contextual information, and improve the accuracy and consistency of the comparison results.

[0132] Please refer to Figure 5 In a feasible implementation manner, step A100 may include steps A110 to A130:

[0133] Step A110, calculating the semantic similarity of each chapter of the document to be compared according to a semantic analysis algorithm to obtain a similarity value.

[0134] Step A120, determining whether the similarity value is greater than a preset similarity threshold.

[0135] Step A130: If it is greater than, the corresponding chapters are paired to obtain paired chapters.

[0136] In this embodiment, first, two or more documents to be compared are obtained from a source such as a file, a database or a network. Each document is divided into different chapters according to the structure of the document (such as title, chapter mark, etc.), and a chapter list of each document is obtained. Secondly, the semantic similarity score of the chapter content is calculated based on a pre-trained semantic analysis model, such as BERT, GPT series, Word2Vec, etc. The similarity score of each pair of chapters is compared with a preset similarity threshold. If the similarity score is greater than or equal to the threshold, the two chapters are considered similar and paired.

[0137] In this embodiment, the semantic similarity between chapters is calculated by a semantic analysis algorithm, which can capture the semantic information in the document, avoid the influence caused by format, structure differences, language and grammar, and obtain parts with substantially similar content.

[0138] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 The document comparison method may further include steps B100 to B400:

[0139] Step B100, during the difference comparison process, detecting the number of characters in the target segment corresponding to the target subtask;

[0140] Step B200: When the number of characters in the target segment is greater than or equal to a preset second character number threshold, the execution of the target subtask is terminated.

[0141] In this embodiment, when executing the target subtask of document difference comparison, the number of characters processed by the subtask is monitored in real time. Whenever a character or a text is processed, the number in the character counter increases accordingly. When the value of the character counter reaches the preset second character number threshold, an exception is thrown, an interrupt signal is sent, or the task status is modified to interrupt the current processing and save the unprocessed part.

[0142] Step B300: Obtain unprocessed document segments corresponding to the number of characters exceeding the second character count threshold.

[0143] Step B400, creating an unprocessed subtask for the unprocessed document fragment, and performing a difference comparison on the unprocessed subtask using the large model.

[0144] In this embodiment, after the task is terminated, the character position or index at the time of interruption is recorded, and the remaining document content is extracted from this position, so as to obtain the unprocessed document content corresponding to the second character number threshold. For the obtained unprocessed document fragment, a new unprocessed subtask is created, and the unprocessed subtask is stored in the task queue, waiting for subsequent processing and execution. After the unprocessed subtask is processed, the previously created unprocessed target subtask needs to be executed. By dynamically monitoring the number of document characters in the target subtask being executed, it can be ensured that when the number of characters currently input is close to the maximum limit of the input length for the large model, the processing of the current target subtask is automatically interrupted or terminated to avoid exceeding the limit.

[0145] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 7 , step S200 also includes steps D100 to D300:

[0146] Step D100, extracting the target subtasks in order from the smallest to the largest number of characters in the documents corresponding to the difference comparison subtasks;

[0147] Step D100, assigning the target subtasks to different processing processes in sequence according to the order in which the target subtasks are extracted;

[0148] Step D100, after being assigned to the last processing process, returning to the first processing process for assignment until the target subtask is completely assigned.

[0149] In this embodiment, the document corresponding to the difference comparison subtask is read and the number of characters is calculated, and the difference comparison subtasks are sorted in order of the number of characters from small to large. Multiple processing processes are created, and each thread will be responsible for processing the comparison task of a part of the documents. An index or counter is used to poll and assign tasks to threads. Each processing process reads the corresponding document and compares it according to the assigned document index list. Threading.Event, threading.Barrier, threading.Lock or other synchronization mechanisms are used to wait for all processing processes to complete the target subtask.

[0150] For example, there are 6 target subtasks and 3 processing processes, and the corresponding character numbers are 500, 1000, 1500, 2000, 2500 and 3000 respectively. The sorted document order is: [Document 1, Document 2, Document 3, Document 4, Document 5, Document 6]. Create 3 threads and assign tasks in a round-robin manner: Thread 1: Document 1 (500 characters), Document 4 (2000 characters); Thread 2: Document 2 (1000 characters), Document 5 (2500 characters); Thread 3: Document 3 (1500 characters), Document 6 (3000 characters).

[0151] In this embodiment, by assigning tasks with similar character numbers to different processing processes, it is ensured that the load of each thread is relatively balanced, and the assigned target subtasks can be completed at the same time, thereby reducing the waiting time.

[0152] The present application provides a document difference comparison device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document difference comparison method in the above-mentioned embodiment one.

[0153] Reference below Figure 8 , which shows a schematic diagram of the structure of a document difference comparison device suitable for implementing the embodiment of the present application. The document difference comparison device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The document difference comparison device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0154] like Figure 8As shown, the document difference comparison device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the document difference comparison device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the document difference comparison device to communicate with other devices wirelessly or by wire to exchange data. Although the document difference comparison device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0155] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0156] The document difference comparison device provided by the present application adopts the document difference comparison method in the above embodiment, which can solve the technical problem of how to improve the efficiency of difference analysis of long documents. Compared with the prior art, the beneficial effects of the document difference comparison device provided by the present application are the same as the beneficial effects of the document difference comparison method provided by the above embodiment, and other technical features in the document difference comparison device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0157] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0158] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0159] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the document difference comparison method in the above-mentioned embodiment.

[0160] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0161] The computer-readable storage medium may be included in the document difference comparison device; or may exist independently without being assembled into the document difference comparison device.

[0162] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the document difference comparison device, the document difference comparison device: obtains the difference comparison subtask corresponding to the document to be compared, and adds each of the difference comparison subtasks to the multi-threaded task queue according to the priority of each of the difference comparison subtasks; extracts at least two target subtasks from the multi-threaded task queue, and in the processing process corresponding to the extracted target subtasks one by one, synchronously performs difference comparison on the target subtasks according to the preset large model to obtain a first difference result; after all the target subtasks are executed, obtains each of the first difference results, and integrates the first difference results to obtain a difference report.

[0163] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0166] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned document difference comparison method, and can solve the technical problem of how to improve the efficiency of difference analysis of long texts. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the document difference comparison method provided by the above-mentioned embodiment, and will not be repeated here.

[0167] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A document difference comparison method, characterized in that: The method includes: Obtaining the difference comparison subtasks corresponding to the documents to be compared, and adding each of the difference comparison subtasks to a multi-threaded task queue according to the priority of each of the difference comparison subtasks; Extracting at least two target subtasks from the multi-threaded task queue, and in a processing process corresponding to the extracted target subtasks one by one, synchronously performing a difference comparison on the target subtasks according to a preset large model to obtain a first difference result; After all the target subtasks are executed, each of the first difference results is obtained, and the first difference results are integrated to obtain a difference report.

2. The method according to claim 1, characterized in that Before the step of obtaining the difference comparison subtasks corresponding to the to-be-processed documents and adding each of the difference comparison subtasks to the multi-threaded task queue according to the priority of each of the difference comparison subtasks, the method further includes: Acquire the document to be compared, and perform chapter matching on the document to be compared according to semantic similarity to obtain paired chapters; Determine a slicing point in the paired chapters, and obtain the number of characters of the segment corresponding to the slicing point; If the number of characters is less than or equal to a preset first character number threshold, determining the slicing point as a target slicing point; According to the target slicing points, each of the paired chapters is segmented to obtain a target segment, and the difference comparison subtask is created for each of the target segments.

3. The method according to claim 2, characterized in that The step of obtaining the document to be compared and matching the sections of the document to be compared according to semantic similarity to obtain paired sections includes: Calculating the semantic similarity of each chapter of the document to be compared according to a semantic analysis algorithm to obtain a similarity value; Determining whether the similarity value is greater than a preset similarity threshold; If it is greater, the corresponding chapters are paired to obtain paired chapters.

4. The method according to claim 1, characterized in that The step of extracting at least two target subtasks from the multi-threaded task queue, and synchronously performing a difference comparison on the target subtasks according to a preset large model in a processing process corresponding to the extracted target subtasks, to obtain a first difference result includes: Constructing a prompt engineering template, wherein the prompt engineering template includes: comparison instructions and output format; Input the target subtask and the prompt engineering template into the preset large model to compare and analyze the differences; Based on the preset large model, the first difference result that conforms to the construction prompt engineering template is generated.

5. The method according to claim 1, characterized in that The method further comprises: During the difference comparison process, detecting the number of characters in the target segment corresponding to the target subtask; When the number of characters in the target segment is greater than or equal to a preset second character number threshold, execution of the target subtask is terminated.

6. The method according to claim 5, characterized in that After the step of terminating the execution of the target subtask when the number of characters of the target segment is greater than or equal to a preset second character number threshold, the method further includes: Obtaining unprocessed document segments exceeding the second character count threshold; An unprocessed subtask is created for the unprocessed document fragment, and the large model performs a difference comparison on the unprocessed subtask.

7. The method according to claim 1, characterized in that Before the step of obtaining the difference comparison subtasks corresponding to the to-be-compared documents and adding each of the difference comparison subtasks to the multi-threaded task queue according to the priority of each of the difference comparison subtasks, the following steps are included: Obtaining the number of target subtasks being executed, and determining whether the number is less than a preset execution number threshold; If it is less than, the difference comparison subtask corresponding to the to-be-compared document is obtained, and each difference comparison subtask is added to the multi-threaded task queue according to the priority of each difference comparison subtask.

8. The method according to claim 1, characterized in that The step of extracting at least two target subtasks from the multi-threaded task queue and processing the extracted target subtasks in a one-to-one correspondence with each other comprises: Extracting the target subtasks in order of the number of characters in the documents corresponding to the difference comparison subtasks from small to large; Allocating the target subtasks to different processing processes in sequence according to the order in which the target subtasks are extracted; After being assigned to the last processing process, the target subtask returns to the first processing process for assignment until all the target subtasks are assigned.

9. A document difference comparison device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document difference comparison method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document difference comparison method according to any one of claims 1 to 8 are implemented.

Images