A file comparison method, apparatus, and medium

Abstract

The application discloses a file comparison method and device and a medium, and relates to the technical field of data processing. The layout analysis model is used for analyzing and processing each target file to obtain corresponding text information, and the text recognition model is used for recognizing and processing the text information to obtain corresponding file titles, so that the search efficiency for searching for the file titles in the files is improved. The file titles are processed in layers to construct a structured outline directory, and target file titles for subsequent title vector processing are obtained, so that the outline comparison and analysis function between the target files is realized through the file titles as file elements. The semantic vector model is used for processing the target file titles to obtain corresponding title vectors, so that the comparison quality reduction caused by the comparison of similar meaning texts by human subjectivity is avoided. The title vector parameters of the target file titles in the outline directory are used for similarity processing, the semantics of the whole title is focused on, and the overall difference is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a file comparison method, device and medium. Background Art

[0002] Conventional document comparison, such as in the bidding and bid evaluation process, uses manual comparison to screen whether there are similarities in the content of different bid documents. Since manual comparison is highly subjective, the overall differences reflected in the bid documents are poor, and the efficiency of the comparison process is reduced.

[0003] Therefore, how to improve the efficiency of file comparison and the overall difference is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of the present invention is to provide a file comparison method, device and medium to solve the problems of poor overall difference and reduced work efficiency caused by conventional manual comparison or comparison with duplicate detection tools.

[0005] To solve the above technical problems, the present invention provides a file comparison method, comprising:

[0006] Obtain each target file; and parse each target file through a layout parsing model to obtain text information of each target file;

[0007] Recognize the text information using a text recognition model to obtain a corresponding file title; and perform layered processing on the file title to obtain a target file title corresponding to the outline directory;

[0008] Processing the target file title using a semantic vector model to obtain a corresponding title vector;

[0009] Based on the similarity processing of each title vector of the current target file and all the title vectors of the remaining target files, the consistency result of the outline directory is obtained.

[0010] On the one hand, after obtaining the consistency results of the outline catalog, it also includes:

[0011] If the consistency results of the outline directories of the current target file and the remaining target files do not exceed the first threshold, the text information is recognized and processed by the text recognition model to obtain the corresponding file text; and the file text is processed by the semantic vector model to obtain the corresponding text vector;

[0012] The text vectors are constructed and processed by an index construction method to obtain a text index library; and the corresponding text segment information is obtained by searching and processing the preset key information vectors in the text index library; wherein each preset key information vector is obtained by processing each preset key information obtained in advance using the semantic vector model;

[0013] The re-ranking model is used to perform similarity processing on each preset key information vector and the corresponding text segment information to obtain the corresponding text segment similarity result to determine the actual key information;

[0014] A vector similarity process is performed based on the actual key information of the current target file and the actual key information of the remaining target files to obtain a corresponding first similarity result, so as to determine the consistency result of the file text.

[0015] On the other hand, after obtaining the consistency result of the file text, it also includes:

[0016] If the consistency results of the file texts of the current target file and the remaining target files do not exceed the second threshold, then the target files are parsed using the layout parsing model to obtain file images of the target files;

[0017] Processing the file image using a picture vector model to obtain a corresponding picture vector;

[0018] Based on the similarity processing of each image vector of the current target file and all image vectors of the remaining target files, the consistency result of the file image is obtained.

[0019] On the other hand, after obtaining the consistency result of the file image, it also includes:

[0020] If the consistency results of the file images of the current target file and the other target files do not exceed the third threshold, then pre-acquire the information byte length corresponding to the file business information;

[0021] If the information byte length is greater than the first preset byte length, the pre-trained model is used to extract the file text of each target file to obtain the actual file business information;

[0022] If the information byte length is less than or equal to the first preset byte length, extracting the file text of each target file using a named entity recognition model to obtain actual file business information;

[0023] Match and compare each actual business information of the current target file with each actual business information of the remaining target files to determine the number of identical types of actual business information;

[0024] The consistency result of the business information of the current target file and the other target files is determined according to the number of the same types of actual business information and the number of the types of actual file business information.

[0025] On the other hand, the file titles are hierarchically processed to obtain target file titles corresponding to the outline directory, including:

[0026] Get the root title and subtitles in each file title;

[0027] Establishing a first mapping relationship between the root title and the corresponding subtitle to determine the outline directory;

[0028] In the outline catalog, if there are at least two sub-topics in the first mapping relationship belonging to the same root title, then the sub-topics corresponding to the same root title and the corresponding root title are taken as a title subset;

[0029] Each of the title subsets is used as the target file title.

[0030] On the other hand, the consistency results of the outline directory are obtained by performing similarity processing on each title vector of the current target file and all the title vectors of the remaining target files, including:

[0031] Determine the first file and the second file corresponding to the pairwise comparison files in each of the target files;

[0032] Taking the first file as a base file, performing similarity processing on each title vector of the base file and all title vectors of the second file in sequence to obtain a first consistency result between the first file and the second file;

[0033] Taking the second file as a new basic file, performing similarity processing on each title vector of the new basic file and all title vectors of the first file in sequence to obtain a second consistency result between the second file and the first file;

[0034] The first consistency result and the second consistency result are averaged to obtain the final consistency result of the first file and the second file.

[0035] On the other hand, before the target file title is processed by using the semantic vector model to obtain the corresponding title vector, the method further includes:

[0036] Using the text recognition model to perform recognition processing on the text information to obtain the corresponding file text;

[0037] Obtaining the file title and file text of the current target file to establish a mapping relationship between the file title and the file text; wherein a file title has at least one sub-file text;

[0038] Input each sub-file text corresponding to each file title in the mapping relationship into the pre-trained language model to output a matching score for each sub-file text corresponding to each file title;

[0039] Counting the number of sub-file texts whose matching scores corresponding to each file title are greater than a first preset score;

[0040] Determining a relevance score between each file title and the sub-file text according to the first number and the number of sub-file texts corresponding to each file title;

[0041] When the relevance score between each file title and the sub-file text exceeds the first preset score, it is determined that the degree of matching between the file title and the file text of the current target file meets the preset requirements, and the step of using the semantic vector model to process the target file title to obtain the corresponding title vector is entered.

[0042] On the other hand, the training process of the pre-trained language model includes:

[0043] Get the training files and initial pre-trained language model;

[0044] Parsing the training file using a layout parsing model to obtain corresponding training text information;

[0045] Recognize the training text information using the text recognition model to obtain the corresponding training file title and training file text;

[0046] Create positive sample data by combining the current training file title and the corresponding first training file text;

[0047] Creating negative sample data using the second training file text corresponding to the current training file title and the remaining training file titles except the current training file title;

[0048] Inputting the positive sample data and the negative sample data into the initial pre-trained language model to output corresponding label results;

[0049] If the label result meets the training requirements or the current number of iterations reaches the iteration requirement, the model parameters of the initial pre-trained language model are fine-tuned to determine the final pre-trained language model.

[0050] In order to solve the above technical problems, the present invention further provides a file comparison device, comprising:

[0051] memory for storing computer programs;

[0052] A processor is used to implement the steps of the file comparison method when executing the computer program.

[0053] To solve the above technical problems, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the file comparison method as described above are implemented.

[0054] The beneficial effect of the present invention is that first, each target file needs to be parsed and processed by a layout parsing model to obtain the corresponding text information, and then the text information is identified and processed by a text recognition model to obtain the corresponding file title. The present invention obtains the file title by processing the two models, which improves the search efficiency of searching for the file title in the file compared to the search for the file title in the manual comparison process. Secondly, the file title is hierarchically processed to construct a structured outline directory, and the target file title for subsequent title vector processing is obtained to realize the outline comparison and analysis function between the target files through the file title as a file element. The target file title is processed using a semantic vector model to obtain the corresponding title vector. Compared with the word comparison, character comparison or phrase comparison performed in the manual comparison process, the comparison quality is reduced due to the subjective comparison of texts with similar meanings. Finally, the title vector parameters of the target file title of the outline directory are used for similarity processing, focusing on the semantics of the entire title, which can capture the theme and overall meaning of the title, reflect the context and semantic structure of the title and the outline directory, and improve the overall difference.

[0055] Secondly, the target file titles are collated to facilitate the subsequent formation of corresponding title vectors. The target file titles form corresponding outline directories, improving the accuracy of overall difference comparisons. Based on the consistency results between different target files and the remaining target files, the text title differences between each target file are determined. The lower the consistency score corresponding to the difference, the greater the difference between the target files. Difference analysis is performed based on the text title element. The file texts of each target file are compared to determine the corresponding consistency results. Each target file is used as the current target file, and a traversal comparison of all key information is performed. During the comparison of key information extracted from the file text, different models are used for comparison analysis corresponding to each element in the file comparison process to improve the accuracy of the key information comparison of the file text. Based on each target file as the current target file, a traversal comparison of all file images is performed. The comparison of different models under the file elements of the file images further improves the accuracy of the image relevance analysis. The consistency results are determined by comparing the business information between multiple target files. The comparison of different models under the file elements of the business information uses a character comparison method to improve the matching accuracy. The title and text of each target file are analyzed and processed to determine the corresponding results for each target file. This avoids mismatches between the text and the title. Consistency analysis of the file's title and text ensures consistency between the file's content and title. The pre-trained language model training process, combined with the positive and negative sample data established for file content and title, allows for optimal application in consistency analysis and comparison of target files, ensuring subsequent comparison accuracy.

[0056] In addition, the present invention also provides a file comparison device and a medium, which have the same beneficial effects as the above-mentioned file comparison method. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 A flowchart of a file comparison method provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of text relevance analysis provided by an embodiment of the present invention;

[0060] Figure 3A schematic diagram of a comparison process between a file title and a file text provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of file element parsing of each target file provided by an embodiment of the present invention;

[0062] Figure 5 A structural diagram of a file comparison device provided by an embodiment of the present invention;

[0063] Figure 6 A structural diagram of a file comparison device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] The core of the present invention is to provide a file comparison method, device and medium to solve the problems of poor overall difference and reduced quality and efficiency caused by conventional manual comparison or comparison with duplicate checking tools.

[0066] In order to enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0067] Document comparison, such as bid comparison, involves detailed comparison of bid documents to ultimately select the best bidder. Conventional bid comparisons can involve collusion with tendering and bidding entities to inflate or depress bid prices, excluding other bidders through price alignment or consistency. Such bid-rigging and bid-rigging practices seriously disrupt fair competition in the market, directly harm the legitimate rights and interests of other bidders, and place honest businesses at a competitive disadvantage. Furthermore, bid-rigging and bid-rigging often result in winning bids not based on strength and credibility, thus impacting the quality and effectiveness of tender projects. During the bid evaluation process, tenderers generally conduct a detailed review of bid documents. By comparing bid documents for originality, consistency in bid content and proposal details, and for conflicts in information such as bidder names, official seals, and product manufacturer authorizations, tenderers can effectively maintain fairness in the bidding process, improve bid quality, and avoid losses for tenderers. Existing manual bid comparison methods are highly subjective, unable to comprehensively assess differences between bids, and are labor-intensive and inefficient.

[0068] Existing bid comparison methods mostly rely on manual comparison for specific information. This involves carefully reading the bidding documents and comparing bidder names, official seals, and other information to determine if there are any conflicts; if there are any similarities between different bids; and if the business information in the bids is consistent. This manual comparison method is highly subjective, unable to comprehensively grasp the differences between bids, and is labor-intensive and inefficient. The document comparison method provided by the present invention can address these technical issues.

[0069] Figure 1 A flowchart of a file comparison method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0070] S11: Obtain each target file; and parse each target file using a layout parsing model to obtain text information of each target file;

[0071] S12: Recognize the text information using a text recognition model to obtain a corresponding file title; and perform layered processing on the file title to obtain a target file title corresponding to the outline directory;

[0072] S13: Processing the target file title using the semantic vector model to obtain a corresponding title vector;

[0073] S14: performing similarity processing on each title vector of the current target file and all title vectors of the remaining target files in turn to obtain consistency results of the outline directory.

[0074] Specifically, the file comparison scenario can be based on comparing a new file that needs to be compared with an existing standard file, or it can be based on comparing all new files that need to be compared with each other, etc., which is not limited here and can be set according to actual conditions. Regarding the acquisition process of the target file, it can be the comparison of bidding documents, the comparison of various specifications, or the comparison and duplication of papers, etc. As long as the file involves the comparison and duplication checking scenario, the file comparison method of the present invention can be used.

[0075] The layout parsing model parses each target file to obtain its text information. This can be done using a model or by identifying and extracting elements based on predefined rules, such as identifying titles and body text based on features like font size, position, and margins. Other methods include template matching, optical character recognition, natural language processing, and graph-, region-, and feature-based approaches. These methods are not limited here and can be set based on actual circumstances.

[0076] The parsing process of the layout parsing model is to obtain the file format attributes of each target file; if the file format attributes are non-image formats, convert each page of each target file into a file corresponding to the image format; perform box selection and annotation processing on each converted target file in the form of text to obtain first annotation data; obtain a multimodal pre-training model; and input the first annotation data of each target file into the multimodal pre-training model to obtain the text information of each target file. It should be noted that text information includes the specific content of the text and the location of the text. Taking the tender document as an example, after receiving the tender document, the tenderer converts each page of the document with a non-image format attribute, such as a Portable Document Format (PDF) format or a Word format, into a file in the image format. Using the annotation tool, the different elements in the image are box selected and annotated on the target file converted into an image to obtain the first annotation data. The annotated first annotation data, including the original image, the coordinates of each element, and the corresponding labels (such as tables and titles), is processed using a multimodal pre-training model (such as the Document AI Multimodal Pre-training Model (LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking) model), whose output parameters correspond to the text information of each target file.

[0077] It should be noted that in this embodiment, the model parameters corresponding to the multimodal pre-trained model can be fine-tuned based on the annotated data in this embodiment to adapt the model parameters corresponding to the target file in this embodiment. The multimodal pre-trained model is used to perform layout parsing to determine the text information of each target file to improve the accuracy of the layout parsing, thereby improving the accuracy of subsequent comparison results.

[0078] In step S12, the text information is recognized and processed using a text recognition model to obtain the corresponding file title. The text recognition model, such as an optical character recognition (OCR) model, or other text recognition models, recognizes both the text content and the text location information to obtain the file title of each target file. The file titles are then layered to obtain the target file titles corresponding to the outline directory, and a structured outline directory is constructed based on the labels. It should be noted that in a target file, there are multiple levels of titles, which can be connected using spaces. The storage format is: [first_title, first_title+" "+second_title, first_title+" "+second_title+" "+third_title, ...]. Spaces are used to distinguish different file titles to avoid errors in the result comparison process caused by multiple file titles being close together.

[0079] It should be noted that the hierarchical processing herein corresponds to the outline directory corresponding to the relationship between the root title and subtitles set in each file title. For subsequent vectorization processing, in some embodiments, the file title is hierarchically processed to obtain the target file title corresponding to the outline directory, including:

[0080] Get the root title and subtitles in each file title;

[0081] Establishing a first mapping relationship between the root title and the corresponding subtitle to determine the outline directory;

[0082] In the outline catalog, if there are at least two sub-topics in the first mapping relationship belonging to the same root title, then the sub-topics corresponding to the same root title and the corresponding root title are taken as the title subset;

[0083] Each subset of titles is used as the target file title.

[0084] Specifically, a first mapping relationship is established between the root title and the corresponding subtitle within each file title, thereby forming multiple first mapping relationships. For example, Title 1, Title 1.1, and Title 1.2, where Title 1.1 and Title 1.2 belong to the same root title (Title 1), correspond to two first mapping relationships and two corresponding title subsets. Therefore, these two title subsets are used as target file titles to facilitate subsequent vectorization of the title vectors.

[0085] The target file title sorting process provided in this embodiment facilitates the subsequent formation of corresponding title vectors, and the target file titles form corresponding outline directories to improve the accuracy of overall difference comparison.

[0086] The target file title is processed using a semantic vector model to obtain a corresponding title vector. It should be noted that the semantic vector model in this embodiment can be a vector model (Embedding Vector Model, EMB) or other semantic vector models, which is not limited here. Compared with the conventional word vector model, the semantic vector model here not only focuses on the semantics of a single word, but also considers the meaning of the word in a specific context and the combination relationship between words, and can more comprehensively represent the semantic information of the text. It captures the changes in the meaning of words in different contexts and better handles semantic relationships such as polysemy, synonyms, and antonyms, thereby performing well in tasks that require deep semantic understanding, such as question-answering systems, text implications, etc. The vector processing method is processed according to the actual semantic vector model selected, and will not be elaborated here.

[0087] In step S14, each title vector based on the current target file is processed similarly with all the title vectors of the remaining target files to obtain a consistency result of the outline directory. It should be noted that in this embodiment, the current target file needs to be compared with all the remaining target files except the current target file to determine the consistency result of the outline directory between the current target file and each of the remaining target files.

[0088] In some embodiments, the consistency result of the outline directory is obtained by performing similarity processing on each title vector of the current target file and all title vectors of the remaining target files in sequence, including:

[0089] Determine the first file and the second file corresponding to the pairwise comparison files in each target file;

[0090] Taking the first file as the base file, performing similarity processing on each title vector of the base file and all title vectors of the second file in sequence to obtain a first consistency result between the first file and the second file;

[0091] Taking the second file as a new basic file, performing similarity processing on each title vector of the new basic file and all title vectors of the first file in sequence to obtain a second consistency result between the second file and the first file;

[0092] The first consistency result and the second consistency result are averaged to obtain the final consistency result of the first file and the second file.

[0093] Specifically, the similarity comparison of files in this embodiment is based on the comparison of pairwise target files. Taking the first file and the second file as an example, the first file is used as the basic file, and each title vector of the basic file is sequentially processed with all the title vectors of the second file for similarity to obtain the first consistency result of the two files.

[0094] The initialization process is as follows:

[0095] scoreA=0 #Based on Bid A, calculate the outline similarity score with Bid B;

[0096] scoreB=0 #Calculate the outline similarity score of Bid B with Bid A based on Bid B;

[0097] title_emb_A={} #Initialize the outline of tender A <title: vector>;

[0098] title_emb_B={} #Initialize the outline of tender B <title: vector>;

[0099] sim_yz=0.9 #Initialize the similarity threshold, adjustable.

[0100] Get the vectors of all titles of Bid A and Bid B;

[0101] for title in biaoshu_A_titles: #Traverse all titles of bid A:

[0102] title_emb_A{title}= emb_model.encode(title);

[0103] for title in biaoshu_B_titles: #Traverse all titles of bid B:

[0104] title_emb_B{title}= emb_model.encode(title).

[0105] When calculating the first consistency result, taking Bid A and Bid B as an example, we calculated the outline consistency score with Bid B based on Bid A.

[0106] # Calculate outline similarity;

[0107] for title_A in biaoshu_A_titles: #Traverse all titles of bid book A;

[0108] #Calculate the similarity between the title of A and all the titles of B;

[0109] sim_scores=[title_emb_A[title_A] @ title_emb_B[title_B].T for title_Bin biaoshu_B_titles];

[0110] #Take the most similar question and calculate the similarity of the subtitles;

[0111] max_sim=math.max( sim_scores) # The place with the highest similarity;

[0112] if( max_sim > sim_yz ):

[0113] scoreA+=max_sim;

[0114] else:

[0115] pass

[0116] scoreA=score / (len(biaoshu_A_titles)).

[0117] The first file 1 (A1, B1, C1, D1) and the second file 2 (A2, B2, C2, D2) are compared. Based on the first file 1, each title of the first file 1 needs to be compared with all the title vectors of the second file 2 to obtain 16 similarities. The similarity between each title and all the title vectors of the second file 2 is screened (through the first threshold screening, that is, 4 similarities are compared with the first threshold. If it exceeds the first threshold, it is determined that the title is similar to a title vector of the second file 2) and recorded as title similarity data. If there are 3 titles that are similar to all the title vectors of the second file 2, the consistency score of the first file 1 and the second file 2 is (3 / 4).

[0118] Similarly, the second file is used as a new basic file, and similarity processing is performed on each title vector of the new basic file and all title vectors of the first file in sequence to obtain a second consistency result between the second file and the first file.

[0119] The first consistency result and the second consistency result are averaged to obtain the final consistency result of the first file and the second file.

[0120] score=(scoreA+scoreB) / 2.

[0121] Among them, score is the consistency score of the outlines of Bid A and Bid B; the lower the score, the greater the difference between the outlines of Bid A and Bid B, the less likely it is to be repeated, and the higher the originality.

[0122] This embodiment provides a determination process based on the consistency results between different target files and other target files to obtain the differences in text titles between target files. If the score of the consistency result corresponding to the difference is lower, it means that the differences between target files are larger, and difference analysis is realized under the element of text title.

[0123] The present invention provides a file comparison method, which obtains target files, parses each target file using a layout parsing model to obtain text information of each target file, recognizes and processes the text information using a text recognition model to obtain corresponding file titles, and hierarchically processes the file titles to obtain target file titles corresponding to an outline directory. The target file titles are processed using a semantic vector model to obtain corresponding title vectors. Based on the similarity between each title vector of the current target file and all title vectors of the remaining target files, a consistency result of the outline directory is obtained. First, each target file needs to be parsed using a layout parsing model to obtain corresponding text information, and then the text information is recognized and processed using a text recognition model to obtain corresponding file titles. The present invention obtains file titles by processing these two models, which improves the efficiency of searching for file titles in files compared to manual comparison. Second, the file titles are hierarchically processed to construct a structured outline directory, and the target file titles are subsequently processed with title vectors, so as to realize the outline comparison and analysis function between target files through the file title element. The target file title is processed using a semantic vector model to generate a corresponding title vector. Compared to manual word, character, or phrase comparisons, this avoids the subjective comparison of texts with similar meanings, which can reduce comparison quality. Finally, similarity processing is performed using the title vector parameters of the target file title from the outline directory. This focuses on the semantics of the entire title, capturing the title's theme and overall meaning, reflecting the context and semantic structure of the title and outline directory, and improving overall differentiation.

[0124] In some embodiments, after obtaining the consistency result of the outline directory, the method further includes:

[0125] If the consistency results of the outline directories of the current target file and the other target files do not exceed the first threshold, the text information is recognized and processed by the text recognition model to obtain the corresponding file text; and the file text is processed using the semantic vector model to obtain the corresponding text vector;

[0126] The text vectors are constructed and processed by an index construction method to obtain a text index library; and the corresponding text segment information is obtained by searching and processing the preset key information vectors in the text index library; wherein each preset key information vector is obtained by processing each preset key information obtained in advance using a semantic vector model;

[0127] The re-ranking model is used to perform similarity processing on each preset key information vector and the corresponding text segment information to obtain the corresponding text segment similarity result to determine the actual key information;

[0128] The actual key information of the current target file is subjected to vector similarity processing with the actual key information of the remaining target files to obtain a corresponding first similarity result, so as to determine the consistency result of the file text.

[0129] Specifically, after the outline directory consistency result is output, based on the outline target consistency result, if the outline directory consistency results of the current target file and the remaining target files do not all exceed the first threshold, that is, the outline directory consistency results of the current target file and the remaining target files are all good, in order to comprehensively compare and ensure that the corresponding comparison result is correct, another file element, the consistency result of the file text, needs to be compared.

[0130] Here, the text information is subjected to recognition processing by the text recognition model, which is similar to the processing mode of recognizing the file title and the title vector by the text recognition model and the semantic vector model in the above embodiment, and will not be described here.

[0131] The text vector is subjected to construction processing by the index construction mode to obtain a text index library. It should be noted that a vector index library is constructed using the file text of the tender document, and the inverted file format (Ivfflat) index method can be used. In the text index library, the corresponding text segment information is obtained by searching according to each preset key information vector. The preset key information vector is obtained by processing each preset key information in advance using the text vector model. The key information needs to be obtained from the file text. The key information is set by the bidder or obtained by feature extraction based on certain rules, or obtained by vector retrieval technology. In this way, the tender documents submitted by the bidders are subjected to text segmentation to obtain a vector set of the tender documents, which can be subjected to vector generation by the EMB model. The topk relevant chunks, i.e., text segment information, are retrieved in the index library.

[0132] The specific text content of the key information example is extracted from the tender document. At this time, the topk relevant contents are retrieved by Ivfflat index. Since the result of this index method has low accuracy, the retrieved chunks and the input key information example are subjected to similarity sorting by a rerank model (Rerank). The similarity processing here, for example, a plurality of text segments are retrieved in the index library, for example, the input "project duration" key information obtains 10 text segments. The 10 text segments are subjected to similarity processing with the preset key information example "project duration" by the rerank model to obtain 10 corresponding similarity results. The text segment corresponding to the maximum similarity is selected as the actual key information.

[0133] It should be noted that the topk detection based on vector index is not as accurate as the result after Rerank sorting. This is related to the vector model and the Rerank model itself; Vector / EMB model: input corpus, generate a vector; input query, generate a vector, and confirm their similarity by calculating the cosine value of the two vectors; Rerank model, input query and corpus, and directly output a similarity; The input of the Rerank model includes query and corpus, and the EMB model either inputs corpus or inputs query, so the Rerank model can more fully compare whether the two are similar.

[0134] For a certain key information, the key information description of Bid A and Bid B, traverse all the key information of Bid A, calculate its similarity with the key information in Bid B in turn, and use the threshold to filter, and finally obtain the correlation score based on Bid A and the key information of Bid B; traverse all the key information of Bid B, calculate its similarity with the key information in Bid A in turn, and use the threshold to filter, and finally obtain the key correlation score based on Bid B and Bid A.

[0135] By traversing all known key information examples, we can obtain multiple scores, and take a weighted average of them to obtain the text relevance score of Bid A and Bid B; the lower the score, the greater the difference between the two annotations and the better the originality.

[0136] Compare the similarity of all the actual key information (project duration, name, and other key information) in Bid A with the similarity of all the actual key information in Bid B. For Bid 1's "Name" and Bid 2's "Name," calculate a similarity score of score1 using the vectors. Similarly, a similarity score of score2 is also calculated for "Plan." The final score is the average score: (score1 + score2) / 2, or the weighted score score1 * w1 + score2 * w2 (weight w1 + w2 = 1). In other words, the vector similarity processing here, based on the actual key information of the current target file, is performed against the actual key information of the remaining target files to obtain the corresponding first similarity result. It should be noted that this only corresponds to the first similarity result for the target files that are compared pairwise. The similarity results for all target files that are compared pairwise can also be obtained, determining the final consistency result for the file text.

[0137] Figure 2 A schematic diagram of a text relevance analysis provided by an embodiment of the present invention is shown in FIG. Figure 2As shown, taking Bid A as an example, all the file texts of Bid A are sliced ​​to obtain multiple chunks, and a vector set is obtained through the EMB model. The vector index is constructed using the Ivfflat indexing method to obtain the topk similar texts. The key information of Bid A is obtained by screening based on the key information examples. Similarly, the key information of Bid B is obtained in the same way. The key information of Bid A and Bid B are compared to obtain the consistency result.

[0138] This embodiment provides a method of comparing the file texts of each target file with each other to determine the corresponding consistency results, taking each target file as the current target file and performing a traversal comparison of all key information. In the comparison process of key information extracted from the file text, different model comparison and analysis are used for the comparison corresponding to each element in the file comparison process to improve the accuracy of the key information of the comparison of the file text.

[0139] In some embodiments, after obtaining the consistency result of the file text, the method further includes:

[0140] If the consistency results of the file texts of the current target file and the remaining target files do not exceed the second threshold, then the layout parsing model is used to parse each target file to obtain a file image of each target file;

[0141] Use the picture vector model to process the file image to obtain the corresponding picture vector;

[0142] Based on the similarity processing of each image vector of the current target file and all image vectors of the remaining target files, the consistency result of the file image is obtained.

[0143] Specifically, considering that the consistency results of the file text are output, based on the consistency results of the file text, if the consistency results of the file text of the current target file and the remaining target files do not exceed the second threshold, that is, the consistency results of the file text of the current target file and the remaining target files are good, in order to conduct a comprehensive comparison and ensure that the corresponding comparison results are correct, it is necessary to perform a consistency result comparison of another file element, the file image.

[0144] For example, bids for bid collusion or bid rigging may use the same structure or architecture diagram, or the same official seal, business license, and other information. For image comparison, the image encoder model (such as the Contrastive Language-Image Pre-training (CLIP) model) is needed to vectorize the image. The image vector model here can be the same as a conventional image vector model, or it can be different, and this is not limited here.

[0145] It should be noted that subsequent similarity processing can be performed directly based on the image vector, or an image index method can be constructed here to speed up the traversal speed. The index construction for the image vector here can be the same as the method of constructing the index library in the comparison process of the consistency results corresponding to the file text in the above embodiment, or it can be different, and there is no limitation here.

[0146] The subsequent image vectors are processed and compared with the similarity of all image vectors of the remaining target files in turn, in the same way as the similarity comparison of the title vectors mentioned above, which is not limited here.

[0147] Taking Bids A and B as examples, we first traverse Bid A's file images, calculate their similarity with those in Bid B, and filter them using a threshold. Ultimately, we obtain a file image relevance score for Bid A and Bid B. We then traverse all Bid B's file images, sequentially calculating their similarity with those in Bid A, and filter them using a threshold. Ultimately, we obtain a file image relevance score for Bid B and Bid A. In this embodiment, the current target file is determined by calculating the file image relevance score for each target file as the current target file during the traversal of all target files.

[0148] The correlation here may be obtained in the same manner as the similarity in the above embodiment, or may be different, which is not limited here.

[0149] The third result is determined based on the mean value of the relevance scores of the file images corresponding to each target file. That is, based on each target file, all the file images of each target file are traversed to calculate the similarity of the file images of the remaining target files. Similarly, conversely, based on the remaining target files, the similarity corresponding to the target file is processed, thereby achieving the authority of the image relevance.

[0150] This embodiment provides a method for traversing and comparing all file images based on each target file as the current target file, and a comparison process of different models under the file elements of the file image, so as to further improve the accuracy of the image relevance analysis.

[0151] In some embodiments, after obtaining the consistency result of the file image, the method further includes:

[0152] If the consistency results of the file images of the current target file and the other target files do not exceed the third threshold, then pre-acquire the information byte length corresponding to the file business information;

[0153] If the information byte length is greater than the first preset byte length, the pre-trained model is used to extract the file text of each target file to obtain the actual file business information;

[0154] If the information byte length is less than or equal to the first preset byte length, the named entity recognition model is used to extract the file text of each target file to obtain the actual file business information;

[0155] Match and compare each actual business information of the current target file with each actual business information of the remaining target files to determine the number of identical types of actual business information;

[0156] The consistency result of the business information of the current target file and the other target files is determined according to the number of the same types of actual business information and the number of the types of actual file business information.

[0157] Specifically, after the consistency results of the file images are output, if the consistency results of the file images of the current target file and the remaining target files do not exceed the third threshold, that is, the consistency results of the file images of the current target file and the remaining target files are all good, in order to fully compare and ensure the corresponding comparison results are correct, it is necessary to compare the consistency results of another file element, the business information. Business information may overlap in the corresponding files. For example, for bids involving collusion / bid rigging, two bids may have the same business information such as legal person, address, email address, and phone number. Based on the comparison of business information, potential collusion / bid rigging risks can be quickly screened out.

[0158] First, business information extraction is performed based on the aforementioned layout parsing method. This requires pre-determining the byte length of the file's morning information. During the extraction process, if the byte length of the information exceeds a predetermined length, the pre-trained Large Language Model (LLM) is used for extraction. If the byte length is less than or equal to the predetermined length, named entity recognition (NER) technology is used to compare and analyze the corresponding business information. Both methods can obtain the actual business information in the file.

[0159] Based on the actual business information of each file in the current target file and the actual business information of each file in the remaining target files, a match is performed to determine the number of identical types of actual business information. During the comparison process, the semantic comparison method cannot be used, and a strict match is required. The code is used for comparison, such as directly comparing two strings / str to see if they are the same. For example, if the phone number of bid a is 222333 and the phone number of bid b is 223333, then the phone numbers of bid a and bid b are considered inconsistent. Only when the two strings are completely consistent are they considered consistent, so it is called a strict match.

[0160] The formula is as follows:

[0161] ;

[0162] Among them, info in the formula A with info B Corresponding to the two bids A and B, the "==" in the middle is the judgment of the two strings (info A with info B ) are exactly equal.

[0163] In other words, the number of contact information within the first N characters needs to be counted. For example, if File 1 and File 2 are compared, if two identical business information characters exist, the corresponding score is (2 / all business information types). The denominator here corresponds to the types of business information, and the numerator corresponds to the number of identical business information types. This means that the consistency of the business information between the current target file and the remaining target files is determined based on the number of identical business information types and the number of business information types in the actual files. The lower the business information consistency score, the less relevant the contact information between the corresponding bids, and the lower the risk of bid collusion or bid rigging.

[0164] The consistency result determined by comparing the business information between multiple target files provided in this embodiment uses a character comparison method to improve the accuracy of matching by comparing different models based on the file elements of the business information.

[0165] In some embodiments, before comparing the target files, it is necessary to check that the content and title of each target file are consistent to avoid mismatching. Therefore, it is necessary to analyze and process the file text and file title of each target file to determine the correlation results of each target file. Before using the semantic vector model to process the target file title to obtain the corresponding title vector, it also includes:

[0166] Use the text recognition model to identify and process the text information to obtain the corresponding file text;

[0167] Obtain the file title and file text of the current target file to establish a mapping relationship between the file title and the file text; wherein a file title has at least one sub-file text;

[0168] Input each sub-file text corresponding to each file title in the mapping relationship into the pre-trained language model to output a matching score for each sub-file text corresponding to each file title;

[0169] Counting the number of sub-file texts whose matching scores corresponding to each file title are greater than a first preset score;

[0170] Determining a relevance score between each file title and the sub-file text according to the first number and the number of sub-file texts corresponding to each file title;

[0171] When the relevance score between each file title and the sub-file text exceeds the first preset score, it is determined that the degree of matching between the file title and the file text of the current target file meets the preset requirements, and the step of using the semantic vector model to process the target file title to obtain the corresponding title vector is entered.

[0172] Specifically, the text recognition model here is the same as the above-mentioned text recognition model and is not limited here. The file title and file text of the current target file are obtained, and a mapping relationship between the file title and the file text is established. A file title has at least one sub-file text. When corresponding to a multi-level file title, the multi-level file title will be connected by spaces to obtain a one-to-one correspondence between the file title and the file text. At the same time, considering that the file text under the file title is long, corresponding to the super-long file text, a segmentation method can be used, that is, multiple chunks of data corresponding to the title are obtained, which are used as a training set.

[0173] Each sub-file text corresponding to each file title within each mapping relationship is input into a pre-trained language model (e.g., Bidirectional Encoder Representations from Transformers (BERT)) to output a matching score for each sub-file text corresponding to each file title. A first number of sub-file texts corresponding to each file title whose matching score exceeds a first preset score is counted. A relevance score between each file title and sub-file text is determined based on the first number and the number of sub-file texts corresponding to each file title. If the relevance score between each file title and sub-file text exceeds the first preset score, the matching degree between the file title and the file text of the current target file is determined to meet a preset requirement. For example, if one title corresponds to 10 texts, the title and each text are paired and input into the BERT model to obtain an output result, i.e., a corresponding score. If the score exceeds the first preset score, the result is filtered out, the first number of sub-file texts that exceed the first preset score is counted, and the first number of sub-file texts that exceed the first preset score is divided by 10 to obtain the relevance score corresponding to the title and the text.

[0174] Calculate the score of the target file's own file title and file text. The formula is as follows:

[0175] ;

[0176] The encoder is a fine-tuned pre-trained model. The numerator corresponds to the sum of the correct result information, that is, the sum of the consistency scores of all file titles and file texts (the probability of predicting a value of 1). The denominator is the sum of the number of data points. The higher the score, the better.

[0177] This embodiment provides an analysis and processing of the file title and file text of each target file to determine the associated results of each target file, which avoids the situation where the text does not match the title, and uses the consistency analysis of the file title and file text of the file itself to ensure the consistency of the file content and title.

[0178] In some embodiments, the training process of the pre-trained language model includes:

[0179] Get the training files and initial pre-trained language model;

[0180] The training file is parsed through the layout parsing model to obtain the corresponding training text information;

[0181] The training text information is recognized and processed by the text recognition model to obtain the corresponding training file title and training file text;

[0182] Create positive sample data by combining the current training file title and the corresponding first training file text;

[0183] Create negative sample data using the second training file text corresponding to the current training file title and the remaining training file titles except the current training file title;

[0184] Input the positive sample data and negative sample data into the initial pre-trained language model to output the corresponding label results;

[0185] If the label result meets the training requirements or the current number of iterations reaches the iteration requirement, the model parameters of the initial pre-trained language model are fine-tuned to determine the final pre-trained language model.

[0186] Specifically, the training file is parsed using a layout parsing model to obtain the corresponding training text information; this text information is based on the text characters and the positional information corresponding to the text content. The training file title and training file text are then identified using a text recognition model. This requires creating positive sample data by combining the current file title of the current training file with the corresponding first training file text; and creating negative sample data by combining the current file title of the current training file with the corresponding second file text of the remaining training file titles. Taking titles 2.1, 2.2, and 2.3 as examples, the positive samples are: <2.1 Title, 2.1 Text>, <2.2 Title, 2.2 Text>, <2.3 Title, 2.3 Text>; the negative samples are: <2.1 Title, 2.2 / 2.3 Text>, <2.2 Title, 2.1 / 2.3 Text>, <2.3 Title, 2.1 / 2.2 Text>. Positive samples: If the title and text content are consistent, the label is 1; negative samples: If the title and text content are inconsistent, the label is 0.

[0187] Combine the positive sample data and the negative sample data into a batch of data, input the data corresponding to the batch into the initial pre-trained model to output the corresponding result information. This corresponds to the constructed training set, which is essentially a two-classification task. It is necessary to use the subject judgment model to convert the pre-trained encoder structure (such as BERT and other structural models) into a good one. Figure 3 A schematic diagram of a comparison process between a file title and a file text is provided in an embodiment of the present invention, such as Figure 3 As shown in the figure, a title and a text chunk are input, and the model is fine-tuned to determine whether the title and text chunk have a corresponding relationship. [cls], [sep], and other special tokens are used to link the title and text chunk. When a positive sample is input into the model, the model outputs a label of 1; when a negative sample is input into the model, the model outputs a label of 0. After a batch is calculated, the model parameters are updated, and the above steps are repeated. When the entire labeled training set is trained, it is considered an epoch. After several epochs, training is complete.

[0188] If the corresponding label result meets the training requirements or the number of iterations reaches a certain value, the model parameters of the initial pre-trained language model are fine-tuned to determine the final pre-trained language model.

[0189] The training process of the pre-trained language model provided in this embodiment is combined with the scenario of positive and negative sample data established by file content and title, so that it can be better applied to the consistency analysis and comparison of the target file itself to ensure the accuracy of subsequent comparison.

[0190] In some embodiments, after determining the consistency of the business information of the current target file and the other target files, the method further includes:

[0191] The current target file is used as the third file, and one of the remaining target files is used as the fourth file;

[0192] Set corresponding weight coefficients for the consistency results of the outline directory, the consistency results of the file text, the consistency results of the file image, and the consistency results of the business information, respectively; wherein the sum of the weight coefficients is 1, and the weight coefficients corresponding to the consistency results of the outline directory, the consistency results of the file text, the consistency results of the file image, and the consistency results of the business information are successively smaller than the weight coefficients corresponding to the consistency results of the previous elements;

[0193] The file consistency results of the third file and the fourth file are obtained by weighted processing based on the consistency results of the outline directories corresponding to the third file and the corresponding weight coefficients, the consistency results of the file texts and the corresponding weight coefficients, the consistency results of the file images and the corresponding weight coefficients, and the consistency results of the business information and the corresponding weight coefficients.

[0194] Specifically, corresponding priority levels and weight coefficients are set based on different results. The associated results here are also based on the comparison process between two files. The consistency results of the outline directory, the consistency results of the file text, the consistency results of the file image and the consistency results of the business information are respectively set with corresponding weight coefficients; wherein the sum of each weight coefficient is 1, and the weight coefficients corresponding to the consistency results of the outline directory, the consistency results of the file text, the consistency results of the file image and the consistency results of the business information are respectively smaller than the weight coefficients corresponding to the consistency results of the previous element; the file consistency results of the pairwise comparison files are obtained by weighted summation.

[0195] The process of determining the final file comparison result provided in this embodiment is based on weighted summation performed by setting corresponding weight coefficients under different results, thereby ultimately improving the accuracy of the comparison.

[0196] Figure 4 A schematic diagram of file element parsing of each target file provided by an embodiment of the present invention is shown as follows: Figure 4As shown, taking a tender document as an example, the tender document is input into the layout parsing model, and the output is the document title, document text, and document image. The document title and document text are obtained through the text recognition model. Outline comparison analysis is used for the document title, and consistency analysis is performed for the document title and document text. For the document text, key information needs to be extracted and text relevance analysis is performed, which uses NER technology to compare and analyze contact information. For the document image, relevance analysis of the document image is performed.

[0197] The above describes in detail various embodiments corresponding to the file comparison method. On this basis, the present invention also discloses a file comparison device corresponding to the above method. Figure 5 This is a structural diagram of a file comparison device provided by an embodiment of the present invention. Figure 5 As shown, the file comparison equipment includes:

[0198] An acquisition module 11 is used to acquire each target file; and parse each target file through a layout parsing model to obtain text information of each target file;

[0199] The first processing module 12 is used to identify the text information using a text recognition model to obtain the corresponding file title; and to perform layered processing on the file title to obtain the target file title corresponding to the outline directory;

[0200] The second processing module 13 is used to process the target file title using the semantic vector model to obtain a corresponding title vector;

[0201] The third processing module 14 is configured to perform similarity processing based on each title vector of the current target file and all title vectors of the remaining target files in sequence to obtain a consistency result of the outline directory.

[0202] Since the embodiments of the device part correspond to the above embodiments, the embodiments of the device part please refer to the description of the embodiments of the method part, and will not be repeated here.

[0203] For an introduction to a file comparison device provided by the present invention, please refer to the above method embodiment, and the present invention will not be described in detail here. It has the same beneficial effects as the above file comparison method.

[0204] Figure 6 A structural diagram of a file comparison device provided by an embodiment of the present invention, such as Figure 6 As shown, the device includes:

[0205] Memory 21, for storing computer programs;

[0206] The processor 22 is configured to implement the steps of the file comparison method when executing a computer program.

[0207] The file comparison device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0208] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented in at least one of the following hardware forms: a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array. The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 22 may also include an artificial intelligence (AI) processor, which is responsible for processing computing operations related to machine learning.

[0209] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the file comparison method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include but is not limited to data involved in the file comparison method, etc.

[0210] In some embodiments, the file comparison device may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0211] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the file comparison device, and may include more or fewer components than shown in the figure.

[0212] The processor 22 implements the file comparison method provided in any of the above embodiments by calling instructions stored in the memory 21 .

[0213] For an introduction to a file comparison device provided by the present invention, please refer to the above method embodiment, and the present invention will not be described in detail here. It has the same beneficial effects as the above file comparison method.

[0214] Furthermore, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor 22, the steps of the above-mentioned file comparison method are implemented.

[0215] It is understood that if the methods in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0216] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above file comparison method.

[0217] Furthermore, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the file comparison method when executed by a processor.

[0218] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above file comparison method.

[0219] The above is a detailed introduction to a file comparison method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0220] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A file comparison method, characterized in that: include: Get each target file; and parsing each of the target files using a layout parsing model to obtain text information of each of the target files; Recognize the text information using a text recognition model to obtain a corresponding file title; and perform layered processing on the file title to obtain a target file title corresponding to the outline directory; Processing the target file title using a semantic vector model to obtain a corresponding title vector; Based on the similarity processing of each title vector of the current target file and all the title vectors of the other target files in turn, the consistency result of the outline directory is obtained; Correspondingly, after obtaining the consistency result of the outline directory, it also includes: If the consistency results of the outline directories of the current target file and the remaining target files do not exceed the first threshold, the text information is recognized and processed by the text recognition model to obtain the corresponding file text; and the file text is processed by the semantic vector model to obtain the corresponding text vector; The text vectors are constructed and processed by an index construction method to obtain a text index library; and the corresponding text segment information is obtained by searching and processing the preset key information vectors in the text index library; wherein each preset key information vector is obtained by processing each preset key information obtained in advance using the semantic vector model; The re-ranking model is used to perform similarity processing on each preset key information vector and the corresponding text segment information to obtain the corresponding text segment similarity result to determine the actual key information; Performing vector similarity processing based on the actual key information of the current target file and the actual key information of the remaining target files to obtain a corresponding first similarity result to determine the consistency result of the file text; Correspondingly, before the target file title is processed by using the semantic vector model to obtain the corresponding title vector, the method further includes: Using the text recognition model to perform recognition processing on the text information to obtain the corresponding file text; Obtaining the file title and file text of the current target file to establish a mapping relationship between the file title and the file text; wherein a file title has at least one sub-file text; Input each sub-file text corresponding to each file title in the mapping relationship into the pre-trained language model to output a matching score for each sub-file text corresponding to each file title; Counting the number of sub-file texts whose matching scores corresponding to each file title are greater than a first preset score; Determining a relevance score between each file title and the sub-file text according to the first number and the number of sub-file texts corresponding to each file title; When the relevance score between each file title and the sub-file text exceeds the first preset score, it is determined that the degree of matching between the file title and the file text of the current target file meets the preset requirements, and the step of using the semantic vector model to process the target file title to obtain the corresponding title vector is entered.

2. The file comparison method according to claim 1, characterized in that: After obtaining the consistency result of the file text, it also includes: If the consistency results of the file texts of the current target file and the remaining target files do not exceed the second threshold, then the target files are parsed using the layout parsing model to obtain file images of the target files; Processing the file image using a picture vector model to obtain a corresponding picture vector; Based on the similarity processing of each image vector of the current target file and all image vectors of the remaining target files, the consistency result of the file image is obtained.

3. The file comparison method according to claim 2, characterized in that: After obtaining the consistency results of the file image, it also includes: If the consistency results of the file images of the current target file and the other target files do not exceed the third threshold, then pre-acquire the information byte length corresponding to the file business information; If the information byte length is greater than the first preset byte length, the pre-trained model is used to extract the file text of each target file to obtain the actual file business information; If the information byte length is less than or equal to the first preset byte length, extracting the file text of each target file using a named entity recognition model to obtain actual file business information; Match and compare each actual business information of the current target file with each actual business information of the remaining target files to determine the number of identical types of actual business information; The consistency result of the business information of the current target file and the other target files is determined according to the number of the same types of actual business information and the number of the types of actual file business information.

4. The file comparison method according to claim 1, wherein: The file titles are hierarchically processed to obtain target file titles corresponding to the outline directory, including: Get the root title and subtitles in each file title; Establishing a first mapping relationship between the root title and the corresponding subtitle to determine the outline directory; In the outline catalog, if there are at least two sub-topics in the first mapping relationship belonging to the same root title, then the sub-topics corresponding to the same root title and the corresponding root title are taken as a title subset; Each of the title subsets is used as the target file title.

5. The file comparison method according to claim 1 or 4, characterized in that: Based on the similarity processing of each title vector of the current target file and all the title vectors of the other target files, the consistency results of the outline directory are obtained, including: Determine the first file and the second file corresponding to the pairwise comparison files in each of the target files; Taking the first file as a base file, performing similarity processing on each title vector of the base file and all title vectors of the second file in sequence to obtain a first consistency result between the first file and the second file; Taking the second file as a new basic file, performing similarity processing on each title vector of the new basic file and all title vectors of the first file in sequence to obtain a second consistency result between the second file and the first file; The first consistency result and the second consistency result are averaged to obtain the final consistency result of the first file and the second file.

6. The file comparison method according to claim 1, wherein: The training process of the pre-trained language model includes: Get the training files and initial pre-trained language model; Parsing the training file using a layout parsing model to obtain corresponding training text information; Recognize the training text information using the text recognition model to obtain the corresponding training file title and training file text; Create positive sample data by combining the current training file title and the corresponding first training file text; Creating negative sample data using the second training file text corresponding to the current training file title and the remaining training file titles except the current training file title; Inputting the positive sample data and the negative sample data into the initial pre-trained language model to output corresponding label results; If the label result meets the training requirements or the current number of iterations reaches the iteration requirement, the model parameters of the initial pre-trained language model are fine-tuned to determine the final pre-trained language model.

7. A file comparison device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the file comparison method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the file comparison method according to any one of claims 1 to 6 are implemented.

Images