Text duplicate checking method based on audit risk, storage medium and electronic device

Abstract

The invention discloses a text duplicate checking method based on audit risks, a storage medium and an electronic device, and belongs to the technical field of text duplicate checking, and the method comprises the following steps: S1, segmenting an audit document to be subjected to duplicate checking into first text fragments according to slice characters, inputting the first text fragments into a word vector model, and obtaining a first word vector library, generating a vector pair according to the first word vector library and a second word vector library of a historical document in the word vector model, and obtaining a first document based on the vector pair; s2, presetting a keyword based on the first text fragment, inputting the first document and the keyword into a mixed model for analysis to obtain second documents, and calculating semantic similarity between each second document and the keyword; and S3, obtaining a third document based on the semantic similarity between each second document and the keyword, and obtaining a final duplicate checking result based on the third document. The problem that due to the fact that the semantic similarity between the text and the keyword is not obtained and the semantic similarity depends on the target feature statement, the duplicate checking result is missed and misreported is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of text duplication checking, and in particular to a text duplication checking method, storage medium and electronic device based on audit risk. Background Art

[0002] With the rapid development of computer and information technology, the amount of data in various industries has increased significantly. In audit work, due to the large amount of text, it is difficult to identify repeated or similar content in the audit materials, and thus it is difficult to discover potential plagiarism, tampering or false statements. Therefore, there is a greater audit risk. For example, a text duplicate checking method, device, equipment, storage medium and program product with patent number CN117852519A, the method includes: performing feature extraction processing on the text to be checked for duplicates to obtain a target feature sentence of the text to be checked for duplicates, wherein the target feature sentence includes at least one target keyword of the text to be checked for duplicates; matching processing is performed according to the target feature sentence with each preset historical text, and the text duplicate checking result is output. By matching the target feature sentence of the text to be checked for duplicates with each preset historical text, and outputting the text duplicate checking result, the efficiency of text duplicate checking can be improved. However, the above scheme relies on the extraction of target feature sentences. If the extraction of target feature sentences is not accurate or comprehensive enough, it will lead to false positives or missed positives. In addition, it only judges whether the text is repeated by comparing the literal similarity of the target feature sentences. It does not consider the situation where the two texts are expressed differently but their meanings may be exactly the same or highly similar, and thus ignores the semantic information of the text. Summary of the invention

[0003] In view of the problem that the prior art fails to obtain the semantic similarity between text and keywords and relies on target feature sentences, resulting in underreporting and false positives in duplicate checking results, the present invention provides a text overreporting method, storage medium and electronic device based on audit risk, which fully utilizes the word vector model to convert the first text segment in the audit document to be checked for duplicate content into a first word vector library, thereby combining all the first text segments in the audit document to be checked for duplicate content with all the text segments of each historical document to generate a vector pair to obtain the first document, thereby avoiding dependence on target feature sentences, and utilizing the characteristics of high-dimensional semantic representation of the hybrid model to comprehensively capture the meaning and contextual information of the text and keywords, thereby significantly avoiding underreporting and false positives in duplicate checking results, solving the problem of underreporting and false positives in duplicate checking results due to failure to obtain the semantic similarity between text and keywords and relying on target feature sentences, improving the accuracy of overreporting and promoting audit risk prevention and control.

[0004] In order to solve the above technical problems, the present invention provides a text duplication checking method based on audit risk, comprising the following steps: S1: dividing the audit document to be duplicating into first text segments according to the slicer, inputting all the first text segments into a word vector model, and then obtaining a first word vector library, generating a vector pair according to the first word vector library and a second word vector library of each historical document in the word vector model, and obtaining a first document based on the vector pair; S2: Based on the preset keywords of all the first text segments, the first document and the keyword are input into the hybrid model for analysis, a second document is obtained, and the semantic similarity between each second document and the keyword is calculated; S3: Obtain a third document based on the semantic similarity between each second document and the keyword, and obtain a final duplicate checking result based on the third document.

[0005] After adopting the above technical solution, the present invention has the following advantages: The word vector model is used to convert the first text segment in the audit document to be checked for duplicate content into the first word vector library, so that all the first text segments in the audit document to be checked for duplicate content are combined with all the text segments of each historical document to generate a vector pair to obtain the first document, and each text segment of the audit document to be checked for duplicate content is compared with the historical document, which can fully cover all the contents in the document, avoid relying on the target feature sentence, and help to find the repeated content hidden in the non-feature sentence. At the same time, combined with the characteristics of the high-dimensional semantic representation of the hybrid model, the meaning and contextual information of the text and keywords are fully captured, thereby significantly avoiding the omission and false positive of the duplicate checking results, and improving the accuracy of the duplicate checking; The first word vector library generated by the word vector model can be fully compared with the second word vector library of each historical document in the word vector model to generate vector pairs. This method avoids the limitation of focusing only on the whole document or a specific part, thereby achieving a comprehensive duplicate check of the entire document content and improving the comprehensiveness of the duplicate check; Obtaining the third document based on the semantic similarity between each second document and the keyword can achieve accurate screening of highly relevant content, helping auditors quickly locate potential problem areas, reducing unnecessary waste of time and energy, and thus improving work efficiency; The problem of underreporting and false positives in duplicate checking results caused by not obtaining the semantic similarity between text and keywords and relying on target feature sentences is solved.

[0006] Preferably, the S1 comprises: S11: preprocessing the to-be-checked duplicate audit document, and dividing the preprocessed to-be-checked duplicate audit document into first text segments according to the slicer; S12: input all first text segments into the word vector model, and then convert the words in each first text segment into a first word vector, obtain the first text vector corresponding to each first text segment based on all the first word vectors corresponding to each first text segment, obtain the first word vector library of the audit document to be checked for duplicate auditing based on the first text vector corresponding to each first text segment, and generate vector pairs according to the first word vector library of the audit document to be checked for duplicate auditing and the second word vector library of each historical document in the word vector model; S13: Based on the hash function, the minimum hash value of each vector in the vector pair is obtained, and the first document is obtained based on the minimum hash value of each vector in the vector pair. By dividing the audit document to be checked for duplicate content into the first text segment, a fine-grained analysis of the document content can be achieved, which helps to find local repetitive or similar content in the document, rather than just repetition of the entire or larger paragraphs. Through fine-grained analysis and comprehensive comparison, a comprehensive check for duplicate content of the entire document can be achieved to reduce omissions. The word vector library generated by the word vector model can be efficiently compared with the historical document word vector library in the word vector model. This comparison method avoids the high computational complexity and low efficiency problems in traditional text comparison methods, and improves the efficiency and comprehensiveness of duplicate checking.

[0007] Preferably, the S12 further includes: Based on the slicer, each historical document is divided into second text segments, all second text segments of each historical document are respectively input into the word vector model, and then the words in each second text segment of each historical document are converted into second word vectors, and the second text vector corresponding to each second text segment of each historical document is obtained based on all second word vectors corresponding to each second text segment of each historical document, and the second word vector library of each historical document is obtained based on the second text vector corresponding to each second text segment of each historical document. By generating the second word vector library through the word vector model, the audit document to be checked for duplicate audit can be efficiently compared with the historical documents in the word vector model, avoiding the high computational complexity and low efficiency problems in the traditional text comparison method, and improving the efficiency of duplicate checking.

[0008] Preferably, the S13 includes: S131: obtaining a first minimum hash value of each vector in the vector pair based on a first hash function, and eliminating vector pairs whose first minimum hash values ​​of each vector in the vector pair are unequal, to obtain valid vector pairs; S132: Based on the second hash function, obtain the second minimum hash value of each vector in the valid vector pair, eliminate the valid vector pairs whose second minimum hash values ​​of each vector in the valid vector pair are not equal, obtain the final vector pair, and determine whether the number of final vector pairs corresponding to each historical document is greater than the preset number. The historical document whose number of corresponding final vector pairs is greater than the preset number is the first document. Directly analyzing all vector pairs one by one will consume a lot of computing resources. By using the hash function to eliminate obviously dissimilar vector pairs, the amount of computation for subsequent processing can be significantly reduced. The first hash function may have certain hash conflicts, but by introducing the second hash function for further screening, the impact of such conflicts on the accuracy of the results can be greatly reduced. The two-step hash processing improves the accuracy of the final duplicate checking results.

[0009] Preferably, S2 includes: S21: Preset keywords based on the number of words in all the first text segments, input the first document and the keywords into a hybrid model, and obtain a second document based on the first frequency and the second frequency of the keywords; S22: Calculate the semantic similarity between each second document and the keyword based on cosine similarity. By using preset keywords and hybrid models, the accuracy of document screening is greatly improved, ensuring that the final second document is highly relevant to the keyword. Through cosine similarity calculation, the evaluation process not only focuses on the literal similarity between the document and the keyword, but also goes deeper into the semantic level, thereby significantly avoiding omissions and false positives in the duplicate checking results and improving the accuracy of duplicate checking.

[0010] Preferably, in S21, acquiring the second document based on the first frequency and the second frequency of the keyword includes: Based on the number of second text fragments containing the keyword in each first document, the first frequency of the keyword in each first document is calculated respectively, and the second frequency of the keyword in each first document is obtained based on the number of times the keyword appears in each first document; based on the first frequency of the keyword in each first document and the second frequency of the keyword in each first document, a score of each first document is calculated, and first documents with scores less than a preset score are eliminated to obtain second documents.

[0011] The first frequency measures the extensive distribution of keywords in the document. By counting the number of times a keyword appears in different text fragments, we can understand the penetration of the keyword in the document content and the contextual diversity. The second frequency reflects the importance of the keyword in the document. By combining the first frequency and the second frequency to calculate the score of the document, we can comprehensively evaluate the relevance of each document to the keyword and eliminate documents with scores lower than the preset scores. This helps reduce noise data and improve the accuracy of subsequent processing and analysis. It can quickly and effectively filter out a subset of documents that are highly relevant to the keyword from a large number of original documents, thereby improving the efficiency of document screening and reducing the workload of subsequent duplicate checking.

[0012] Preferably, the S22 includes: S221: constructing a vocabulary based on the second document and the keyword, obtaining a frequency value based on the number of second text segments containing the keyword in the second document, and vectorizing each second text segment in each second document based on the vocabulary and the frequency value; S222: vectorizing the keywords based on the vocabulary, and calculating the cosine similarity between the vectorized keywords and each vectorized second text segment in each second document; S223: The total cosine similarity between all vectorized second text segments and vectorized keywords in each second document is the semantic similarity between each second document and the keyword. By vectorizing text segments and keywords, the semantic similarity between them can be more accurately evaluated. The analysis method based on numerical representation reduces subjectivity and uncertainty, and improves the accuracy of semantic analysis.

[0013] Preferably, in S3, acquiring the third document based on the semantic similarity between each second document and the keyword includes: Marking a second document whose semantic similarity with the keyword is greater than a preset semantic similarity as a clean document; The third document is obtained by eliminating the second text fragment whose cosine similarity between the vectorized second text fragment and the vectorized keyword is less than the preset cosine similarity in each clean document. The documents that are highly relevant to the keywords and whose content is relatively accurate and useful are screened out. At the same time, after determining the clean documents, the second text fragments whose cosine similarity with the keywords is lower than the preset threshold are eliminated, which further refines the document content and ensures that each text fragment in the final third document is closely related to the keyword, thereby improving the overall quality and accuracy of the document. Through the above two-step screening process, the amount of text that needs to be processed during subsequent information extraction, analysis or processing can be significantly reduced, which helps to improve the efficiency of document processing, reduce unnecessary calculations and resource consumption, and significantly improve the quality of the final third document.

[0014] The beneficial effects of this program: The word vector model is used to convert the first text segment in the audit document to be checked for duplicate content into the first word vector library, so that all the first text segments in the audit document to be checked for duplicate content are combined with all the text segments of each historical document to generate a vector pair to obtain the first document, and each text segment of the audit document to be checked for duplicate content is compared with the historical document, which can fully cover all the contents in the document, avoid relying on the target feature sentence, and help to find the repeated content hidden in the non-feature sentence. At the same time, combined with the characteristics of the high-dimensional semantic representation of the hybrid model, the meaning and contextual information of the text and keywords are fully captured, thereby significantly avoiding the omission and false positive of the duplicate checking results, and improving the accuracy of the duplicate checking; The first word vector library generated by the word vector model can be fully compared with the second word vector library of each historical document in the word vector model to generate vector pairs. This method avoids the limitation of focusing only on the whole document or a specific part, thereby achieving a comprehensive duplicate check of the entire document content and improving the comprehensiveness of the duplicate check; By continuously updating documents and then obtaining third documents based on the semantic similarity between each second document and the keyword, it is possible to accurately screen highly relevant content, helping auditors to quickly locate potential problem areas, reducing unnecessary time and energy waste, and thus improving resource utilization and duplicate checking efficiency; The problem of underreporting and false positives in duplicate checking results caused by not obtaining the semantic similarity between text and keywords and relying on target feature sentences is solved.

[0015] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the text duplication checking method based on audit risk are implemented.

[0016] The beneficial effects of this program: Storing the text duplication check program on a computer-readable storage medium does not require a large amount of physical space, saving storage space, while facilitating management and maintenance, and improving management efficiency. Program files on a computer-readable storage medium can be easily shared and transmitted through a network or other media, facilitating collaboration and communication between different users or teams, and improving the flexibility of project management.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the text duplication checking method based on audit risk when executing the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts.

[0019] Figure 1 It is a method flow diagram of the text duplication checking method based on audit risk of the present invention; DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation method described herein is only an optimal embodiment of the present invention, which is only used to explain the present invention and does not limit the scope of protection of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the drawings; the process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0022] Embodiment 1: like Figure 1 As shown in the figure, the text duplication checking method based on audit risk includes the following steps: S1: Divide the to-be-checked duplicate audit document into first text segments according to the slicer, input all the first text segments into the word vector model, and then obtain the first word vector library, generate vector pairs according to the first word vector library and the second word vector library of each historical document in the word vector model, and obtain the first document based on the vector pairs.

[0023] The S1 includes: S11: preprocessing the to-be-checked duplicate audit document, and dividing the preprocessed to-be-checked duplicate audit document into first text segments according to the slicer; S12: input all first text segments into the word vector model, and then convert the words in each first text segment into a first word vector, obtain the first text vector corresponding to each first text segment based on all the first word vectors corresponding to each first text segment, obtain the first word vector library of the audit document to be checked for duplicate auditing based on the first text vector corresponding to each first text segment, and generate vector pairs according to the first word vector library of the audit document to be checked for duplicate auditing and the second word vector library of each historical document in the word vector model; S13: Obtain a minimum hash value of each vector in the vector pair based on a hash function, and obtain a first document based on the minimum hash value of each vector in the vector pair.

[0024] The S12 further includes: Based on the slicer, each historical document is divided into second text segments, all the second text segments of each historical document are respectively input into the word vector model, and then the words in each second text segment of each historical document are converted into second word vectors, and the second text vector corresponding to each second text segment of each historical document is obtained based on all the second word vectors corresponding to each second text segment of each historical document, and the second word vector library of each historical document is obtained based on the second text vector corresponding to each second text segment of each historical document.

[0025] The S13 includes: S131: obtaining a first minimum hash value of each vector in the vector pair based on a first hash function, and eliminating vector pairs whose first minimum hash values ​​of each vector in the vector pair are unequal, to obtain valid vector pairs; S132: Based on the second hash function, obtain the second minimum hash value of each vector in the valid vector pair, eliminate the valid vector pairs whose second minimum hash values ​​of each vector in the valid vector pairs are unequal, obtain final vector pairs, and determine whether the number of final vector pairs corresponding to each historical document is greater than a preset number. The historical document whose number of corresponding final vector pairs is greater than the preset number is the first document.

[0026] The word vector model uses the Chinese version of the text representation model bge-large-zh-v1.5 developed by Zhiyuan Research Institute. It is a Chinese sentence vector model based on a dual encoder architecture and is trained through contrastive learning. It can generate high-quality word vectors. Preprocess the documents to be checked for duplicate auditing, including removing irrelevant information in the documents, such as headers, watermarks, footers, etc., and recursively check the characters to segment the text. First, use symbols such as ",;.!?" as segmentation characters to segment and obtain a set of segmented strings. For the segmented strings, try to combine them from front to back. Under the premise that the length of the combined string is less than 38, the longer the length of the combined string is, the better, and the whole sentence is given priority. Then, several first text segments are obtained, and all the first text segments are input into the word vector model, and then the words in each first text segment are converted into the first word vector. Based on all the first word vectors corresponding to each first text segment, the first text vector corresponding to each first text segment is obtained. Based on the first text vector corresponding to each first text segment, the first word vector library of the documents to be checked for duplicate auditing is obtained. Vector pairs are generated according to the first word vector library of the documents to be checked for duplicate auditing and the second word vector library of each historical document in the word vector model. For example, if the historical document A has two text segments A1 and A2, and the historical document B has three text segments B1, B2, and B3, the second text vectors of A1 and A2 are obtained using the word vector model. Get the second text vector of B1, B2, B3 At this time, the second word vector library of document A is The second word vector library of document B is If the first word vector library of the audit document to be checked for duplicate auditing is The vector pair is In this embodiment, the first hash function is a minimum hash signature algorithm. If the first hash function is used to calculate hash1(A1)=1, hash1(A2)=1, hash1(C2)=1, hash1(C1)=0, hash1(B1)=0, hash1(B2)=0, then the valid vector pair is The second hash function is a locality-sensitive hash filtering algorithm. If the second hash function is used to calculate hash2(A1)=1, hash2(A2)=0, hash2(B1)=0, hash2(B2)=0, hash(C2)=0, hash2(C1)=0, then the final vector pair is If the preset number is 1, the first document is document B. Directly analyzing all vector pairs one by one will consume a lot of computing resources. By using the hash function to eliminate obviously dissimilar vector pairs, the amount of computation for subsequent processing can be significantly reduced. The first hash function may have certain hash conflicts, but by introducing the second hash function for further screening, the impact of such conflicts on the accuracy of the results can be greatly reduced. The two-step hash processing improves the accuracy of the final duplicate checking results.

[0027] S2: Based on the preset keywords of all the first text segments, the first document and the keywords are input into the hybrid model for analysis, the second document is obtained, and the semantic similarity between each second document and the keyword is calculated.

[0028] The S2 includes: S21: Preset keywords based on the number of words in all the first text segments, input the first document and the keywords into a hybrid model, and obtain a second document based on the first frequency and the second frequency of the keywords; S22: Calculate the semantic similarity between each second document and the keyword based on the cosine similarity.

[0029] In S21, acquiring the second document based on the first frequency and the second frequency of the keyword includes: Based on the number of second text fragments containing the keyword in each first document, the first frequency of the keyword in each first document is calculated respectively, and the second frequency of the keyword in each first document is obtained based on the number of times the keyword appears in each first document; based on the first frequency of the keyword in each first document and the second frequency of the keyword in each first document, a score of each first document is calculated, and first documents with scores less than a preset score are eliminated to obtain second documents.

[0030] The S22 includes: S221: constructing a vocabulary based on the second document and the keyword, obtaining a frequency value based on the number of second text segments containing the keyword in the second document, and vectorizing each second text segment in each second document based on the vocabulary and the frequency value; S222: vectorizing the keywords based on the vocabulary, and calculating the cosine similarity between the vectorized keywords and each vectorized second text segment in each second document; S223: The total cosine similarity between all the vectorized second text segments and the vectorized keywords in each second document is the semantic similarity between each second document and the keyword.

[0031] The hybrid model is a combination of the full-text retrieval model and the word vector retrieval model. The full-text retrieval model uses the BM25 algorithm to calculate the similarity between two texts. The core idea of ​​BM25 is based on the probability model and TF-IDF, and some adjustment factors are introduced to improve the retrieval effect. The word vector retrieval model uses cosine similarity as a method to measure the directional similarity of two vectors, calculates the correlation of all text pair vectors in turn, and filters out text pairs with low correlation by setting thresholds. In this embodiment, the keyword can be one word or multiple words. If the audit document to be checked for duplicates consists of two text segments, the number of occurrences of words a, b, c, and d in the first text segment is 3, 4, 2, and 5 respectively, and the number of occurrences of words a, b, c, and d in the second text segment is 2, 3, 1, and 0 respectively, then the keyword is b. If the historical document A consists of two second text segments, the first second text segment is aashdge, and the second second text segment is wqduabc, and the historical document B consists of three second text segments, the first second text segment is uhygji, the second second text segment is asdjbgdh, and the third second text segment is hdubbehsg, then the keyword The first frequency in document A is 1, the second frequency is 1, the first frequency of the keyword in document B is 2, the second frequency is 3, the score of document A is 2, the score of document B is 5, if the preset score is 4, then the second document is document B, the words in the second document are integrated with the keywords to build a vocabulary, and the frequency value is obtained according to the number of second text segments containing the keyword in the second document. If the keyword is b, the historical document B consists of three second text segments, the first second text segment is uhygji, the second second text segment is asdjbgdh, and the third second text segment is hdubbehsg, then the frequency value is 2, and then each second text segment in each second document is vectorized according to the vocabulary and frequency value. If each second text segment of the vectorization of the second document C is The key words for vectorization are By formula Calculate the cosine similarity between the vectorized keywords and each second text segment vectorized in each second document, where: for or or Determine according to demand. and The cosine similarity of for like and The cosine similarities of the second document C are 1, 1, and 0 respectively, then the total cosine similarity of document C is 2, that is, the semantic similarity between the second document C and the keyword is 2. By using preset keywords and hybrid models, the accuracy of document screening is greatly improved, ensuring that the final second document is highly relevant to the keyword. Through the calculation of cosine similarity, the evaluation process not only focuses on the literal similarity between the document and the keyword, but also goes deeper into the semantic level, thereby significantly avoiding omissions and false positives in the duplicate checking results and improving the accuracy of duplicate checking.

[0032] S3: Obtain a third document based on the semantic similarity between each second document and the keyword, and obtain a final duplicate checking result based on the third document.

[0033] In S3, acquiring the third document based on the semantic similarity between each second document and the keyword includes: Marking a second document whose semantic similarity with the keyword is greater than a preset semantic similarity as a clean document; The third document is obtained by eliminating the second text segment whose cosine similarity between the vectorized second text segment and the vectorized keyword is less than a preset cosine similarity in each clean document.

[0034] If each second text segment of the vectorization of the second document C is The key words for vectorization are like like and The cosine similarities of the second document C and the keyword are 1, 1, and 0 respectively. At this time, the semantic similarity between the second document C and the keyword is 2. If the preset semantic similarity is 1, the second document C is a clean document. If the preset cosine similarity is 0, the keywords in the second document C are removed. Get the third document, which only includes and The corresponding second text segment, and The corresponding second text fragment is the similar paragraph of the audit document to be checked for duplicates, and the detection results are then generated, and a file duplicate check report is issued, including the execution time, executor, number of file comparisons, similarity, similar paragraphs, etc. By screening out documents that are highly relevant to the keywords, relatively accurate and useful in content, and after determining the clean documents, the second text fragments whose cosine similarity with the keywords is lower than the preset threshold are eliminated, and the document content is further refined to ensure that each text fragment in the final third document is closely related to the keywords, thereby improving the overall quality and accuracy of the document. Through the above two-step screening process, the amount of text that needs to be processed during subsequent information extraction, analysis or processing can be significantly reduced, which helps to improve the efficiency of document processing and reduce unnecessary calculations and resource consumption.

[0035] In this embodiment, a similarity re-ranking model can also be used. The similarity re-ranking model adopts the Chinese version of the text representation model bge-reranker developed by Zhiyuan Research Institute, and adopts a dual encoder (Bi-Encoder) architecture, one for encoding queries (Query Encoder) and the other for encoding documents (Document Encoder). Each encoder converts the input into a fixed-length vector representation. The model uses a contrastive learning method to train the encoder to maximize the similarity between positive examples and minimize the similarity between negative examples. The similarity between the third document and the audited document to be checked for duplicates is further accurately calculated through the re-ranking model, and the similarity is re-ordered from high to low, and the text fragments above the specified threshold are taken as the final recalled text fragments, thereby improving the accuracy of the duplicate check.

[0036] In this embodiment, the specific operations for checking for duplicate documents are as follows: Step 1: Upload files. File formats such as PDF, Word, and Excel are supported. The files to be checked for duplicates can be compared with historical files for duplicate checking. It also supports uploading multiple files to be checked for duplicates and checking for duplicates among these multiple files. Step 2: File directory identification. The system field identifies the directory structure of the uploaded document. Users can select the content to be checked for duplicate content based on business needs, which plays a role in accurate duplicate checking. Step 3: Robot recognition, all documents are compared pairwise and duplicate content is retrieved; Step 4: Generate the detection results and issue a file duplication check report, including execution time, executor, number of file comparisons, similarity, similar paragraphs, etc.; Step 5: Download the duplicate checking report, supporting functions such as report downloading and saving.

[0037] Embodiment 2: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the text duplication checking method based on audit risk.

[0038] Embodiment 3: An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the text duplication checking method based on audit risk are implemented.

[0039] The specific implementation methods described above are preferred implementation methods of the text duplication checking method, storage medium and electronic device based on audit risk of the present invention, and are not intended to limit the specific implementation scope of the present invention. The scope of the present invention includes but is not limited to the specific implementation methods. All equivalent changes made in accordance with the shape and structure of the present invention are within the protection scope of the present invention.

Claims

1. The text duplication checking method based on audit risk is characterized by: The following steps are involved: S1: Slice the to-be-checked duplicate audit document into first text segments according to the slicer, input all the first text segments into the word vector model, and then obtain a first word vector library, generate a vector pair according to the first word vector library and the second word vector library of each historical document in the word vector model, and obtain the first document based on the vector pair; S2: Based on the preset keywords of all the first text segments, the first document and the keyword are input into the hybrid model for analysis, a second document is obtained, and the semantic similarity between each second document and the keyword is calculated; S3: Obtain a third document based on the semantic similarity between each second document and the keyword, and obtain a final duplicate checking result based on the third document.

2. The text duplication checking method based on audit risk according to claim 1 is characterized in that: The S1 includes: S11: preprocessing the to-be-checked duplicate audit document, and dividing the preprocessed to-be-checked duplicate audit document into first text segments according to the slicer; S12: input all first text segments into the word vector model, and then convert the words in each first text segment into a first word vector, obtain the first text vector corresponding to each first text segment based on all the first word vectors corresponding to each first text segment, obtain the first word vector library of the audit document to be checked for duplicate auditing based on the first text vector corresponding to each first text segment, and generate vector pairs according to the first word vector library of the audit document to be checked for duplicate auditing and the second word vector library of each historical document in the word vector model; S13: Obtain a minimum hash value of each vector in the vector pair based on a hash function, and obtain a first document based on the minimum hash value of each vector in the vector pair.

3. The text duplication checking method based on audit risk according to claim 2 is characterized in that: The S12 also includes: dividing each historical document into second text segments based on the slicer, inputting all the second text segments of each historical document into the word vector model respectively, and then converting the words in each second text segment of each historical document into a second word vector, obtaining the second text vector corresponding to each second text segment of each historical document based on all the second word vectors corresponding to each second text segment of each historical document, and obtaining the second word vector library of each historical document based on the second text vector corresponding to each second text segment of each historical document.

4. The text duplication checking method based on audit risk according to claim 2 is characterized in that: The S13 includes: S131: obtaining a first minimum hash value of each vector in the vector pair based on a first hash function, and eliminating vector pairs whose first minimum hash values ​​of each vector in the vector pair are unequal, to obtain valid vector pairs; S132: Based on the second hash function, obtain the second minimum hash value of each vector in the valid vector pair, eliminate the valid vector pairs whose second minimum hash values ​​of each vector in the valid vector pairs are unequal, obtain final vector pairs, and determine whether the number of final vector pairs corresponding to each historical document is greater than a preset number. The historical document whose number of corresponding final vector pairs is greater than the preset number is the first document.

5. The text duplication checking method based on audit risk according to claim 3 is characterized in that: The S2 includes: S21: Preset keywords based on the number of words in all the first text segments, input the first document and the keywords into a hybrid model, and obtain a second document based on the first frequency and the second frequency of the keywords; S22: Calculate the semantic similarity between each second document and the keyword based on the cosine similarity.

6. The text duplication checking method based on audit risk according to claim 5 is characterized in that: In S21, acquiring the second document based on the first frequency and the second frequency of the keyword includes: Based on the number of second text fragments containing the keyword in each first document, the first frequency of the keyword in each first document is calculated respectively, and the second frequency of the keyword in each first document is obtained based on the number of times the keyword appears in each first document; based on the first frequency of the keyword in each first document and the second frequency of the keyword in each first document, a score of each first document is calculated, and first documents with scores less than a preset score are eliminated to obtain second documents.

7. The text duplication checking method based on audit risk according to claim 5 is characterized in that: The S22 includes: S221: constructing a vocabulary based on the second document and the keyword, obtaining a frequency value based on the number of second text segments containing the keyword in the second document, and vectorizing each second text segment in each second document based on the vocabulary and the frequency value; S222: vectorizing the keywords based on the vocabulary, and calculating the cosine similarity between the vectorized keywords and each vectorized second text segment in each second document; S223: The total cosine similarity between all the vectorized second text segments and the vectorized keywords in each second document is the semantic similarity between each second document and the keyword.

8. The text duplication checking method based on audit risk according to claim 7 is characterized in that: In S3, acquiring the third document based on the semantic similarity between each second document and the keyword includes: Marking a second document whose semantic similarity with the keyword is greater than a preset semantic similarity as a clean document; The third document is obtained by eliminating the second text segment whose cosine similarity between the vectorized second text segment and the vectorized keyword is less than a preset cosine similarity in each clean document.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the text duplication checking method based on audit risk as described in any one of claims 1 to 8 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the text duplication checking method based on audit risk as described in any one of claims 1 to 8 are implemented.

Images