Construction file data alignment processing method, electronic equipment and storage medium

Abstract

The invention discloses a construction file data alignment processing method, electronic equipment and a storage medium, and relates to the technical field of construction file data alignment, and the method comprises the following steps: converting all construction file data into a uniform format, and ensuring the uniform data type; using a data alignment algorithm to match and align the construction file data of different data sources; integrating and merging the aligned construction file data, and screening out a part of reference data as an integration basis; generating a uniform data set by taking the reference data as a standard, and removing repeated data in the data set; the method is used for solving the problems that repeated data is not accurately judged and removed in an existing construction file data alignment technology, and consequently data missing and excessive repeated data are prone to occurring in the construction file.

Description

Technical Field

[0001] The present invention relates to the technical field of construction file data alignment, and in particular to a construction file data alignment processing method, electronic equipment and storage medium. Background Art

[0002] Construction document data alignment technology refers to the integration and matching of construction document data from different sources or formats to ensure that the data is consistent and correctly corresponds in structure and content. This technology usually involves coordinating data from different systems, software or teams to achieve data consistency and integrity in construction projects. Data alignment technology can include processes such as data cleaning, data conversion, data mapping and data association, which aims to eliminate data inconsistencies and conflicts, improve data quality and availability, and thus support the smooth progress and management of construction projects.

[0003] The existing construction file data alignment technology usually converts the construction files into a unified format and then directly merges them, and then removes duplicate data after merging. This method may cause the loss of the content of the original file. In most cases, there are a lot of duplicate contents in the construction files, but this duplicate content serves to recall the previous content and avoid repeated searches by viewers. This method will cause such duplicate contents to be removed. At the same time, the standard for judging duplicate data requires the use of semantic recognition technology, and after semantic recognition, it is necessary to judge whether they are duplicates. The existing construction file data alignment technology usually uses a unified standard for judgment, but based on the length of the sentence, the judgment basis will also change. Using a unified standard for judgment will lead to inaccurate judgment of duplicate data, resulting in a large amount of duplicate data after the construction file is aligned. For example, in the publication number CN 117725118 A's patent application discloses a data alignment method and device. The scheme is to judge the similarity in each data unit through a preset threshold. In the similarity judgment, the length of the sentence will change the similarity judgment base. After the base changes, the threshold will also change. When judging by the preset threshold, sentences of different lengths will cause a large number of deviations in the judgment results. The existing construction file data alignment technology is still not accurate enough in judging and removing duplicate data, which makes construction files prone to data missing and excessive duplicate data. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the prior art to at least a certain extent, by converting all construction file data into a unified format to ensure the uniformity of data types, then screening the data in the file content to screen out the text part and the table part, and then matching and aligning the text part through a data alignment algorithm, matching and aligning the table part and the file table through a data alignment algorithm, integrating and merging the aligned construction file data, screening out a benchmark data as the basis for integration, and finally generating a unified data set based on the benchmark data, and identifying and eliminating duplicate data based on the number of words in the sentence, so as to solve the problem that the existing construction file data alignment technology still has the problem of insufficient accuracy in judging and removing duplicate data, resulting in data missing and excessive duplicate data in construction files.

[0005] To achieve the above objectives, in a first aspect, the present application provides a construction document data alignment processing method, comprising the following steps:

[0006] Convert all construction document data into a unified format to ensure uniform data types;

[0007] Use data alignment algorithms to match and align construction document data from different data sources;

[0008] Integrate and merge the aligned construction document data, and select a benchmark data as the basis for integration;

[0009] Using the benchmark data as the standard, a unified data set is generated and duplicate data is removed.

[0010] Furthermore, converting all construction document data into a unified format to ensure uniform naming and data type of data fields includes the following sub-steps:

[0011] Obtaining construction documents, which include text files, table files, and image files;

[0012] For text files, directly extract the text in the text files and mark the extracted text as file content;

[0013] For table files, directly obtain the entire table and mark the extracted table as a file table;

[0014] For image files, the text in the image files is extracted through image text recognition technology, and the extracted text is marked as file content;

[0015] The data types of the file content and the file table are both text types.

[0016] Furthermore, using a data alignment algorithm, matching and aligning construction document data from different data sources includes the following sub-steps:

[0017] Filter the data in the file content to select the text part and the table part;

[0018] The text part is matched and aligned through the data alignment algorithm, and the table part and the file table are matched and aligned through the data alignment algorithm.

[0019] Furthermore, screening the data in the file content to screen out the text part and the table part includes the following sub-steps:

[0020] The file contents are uniformly saved to a Word document. In the Word document, the table has table attributes and can be directly extracted;

[0021] Extract the table in the file content, mark it as the table part, and mark the remaining file content as the text part;

[0022] The text parts extracted from different construction documents are independent of each other. The text parts are numbered according to the different construction documents. n Indicates that n is a positive integer and n is the sequence number of F.

[0023] Furthermore, matching and aligning the text part by a data alignment algorithm, and matching and aligning the table part and the file table by a data alignment algorithm include the following sub-steps:

[0024] Setting data features, wherein the data features include contract number, date, construction location, participating units and person in charge; the date includes start date and completion date, and the participating units include contractors, subcontractors and supervision units;

[0025] Set delimiters, which include the symbols "colon", "comma", "period", "semicolon", "space" and "newline";

[0026] Find the data feature in the text part, mark it as the text feature, start with the text feature, find the first delimiter and the second delimiter after the text feature, and mark them as the first delimiter and the second delimiter respectively;

[0027] Marking the content between the first partition and the second partition as a text feature value, wherein the text feature value does not include the first partition and the second partition;

[0028] If the text features are the contract number, construction location, and person in charge, then directly compare F nIf the text feature values ​​are the same, a homology signal is output; if the text feature values ​​are different, a non-homology signal is output;

[0029] If the text feature is a date, compare F n Whether the start date and completion date in the text feature values ​​are the same, if they are the same, output a homology signal; if they are different, output a non-homology signal;

[0030] If the text feature is a participating unit, then compare F n Whether the contractors, subcontractors and supervisors in the text feature values ​​are the same, if they are the same, then a homologous signal is output; if they are different, then a non-homologous signal is output;

[0031] If the same source signal is output, the F of the same source signal will be output n The corresponding construction documents are aligned and marked as homologous data;

[0032] The table parts and file tables extracted from the different construction files are independent of each other; the table parts and file tables are collectively referred to as table data, the header names of the table data are extracted, the header names of different table data are compared, and the construction files corresponding to the table data with the same header names are marked as homologous data. The integration object of the homologous data is the construction file, so the homologous data integrated by the text part, the table part and the file table are the same.

[0033] Furthermore, the aligned construction document data is integrated and merged, and a benchmark data is selected as the basis for integration, including the following sub-steps:

[0034] For any homologous data, the text files, table files and image files in the homologous data are collectively referred to as homologous files. The homologous files are numbered and represented by the symbol R. m Represents, where m is a positive integer and m is the sequence number of R;

[0035] With any R m As the base file, other R m Compare the duplicate files with the benchmark files, obtain the duplicate rate, and calculate the different R m The sum of the duplicate check rates obtained through analysis is marked as the duplication index;

[0036] Each R m is the benchmark file, and the duplication checking steps are executed cyclically to obtain max(m) duplication indexes, where max(m) is the maximum value of m;

[0037] Find the maximum value in the repetitiveness index, mark it as the maximum repetitive index, and mark the benchmark file corresponding to the maximum repetitive index as the benchmark data.

[0038] Furthermore, using the benchmark data as a standard, generating a unified data set and removing duplicate data therein includes the following sub-steps:

[0039] For the table parts and file tables in the homologous data, they are collectively referred to as homologous tables. By comparing the table header names, if there are homologous tables with the same header name, only one copy is retained, and the rest of the same homologous tables are all removed;

[0040] For the text part in the homologous data, extract the content between the first line indented to the period in the text part and mark it as a complete sentence, and extract the content between the period and the period and mark it as a complete sentence;

[0041] Each R m Different numbers of complete sentences are obtained by extraction, and the complete sentences extracted from the benchmark data are marked as benchmark sentences, and the remaining complete sentences are marked as merged sentences;

[0042] The m corresponding to the benchmark data is marked as i, and the benchmark statements are numbered and represented by the symbol T(i,j), where j is a positive integer and (i,j) is the sequence number of T, and T(i,j) represents R i The jth benchmark statement in ;

[0043] The merged statements are numbered and represented by the symbol P(m,h), where h is a positive integer and (m,h) is the sequence number of P. P(m,h) represents R m For the h-th merge statement in P(m,h), m in P(m,h) is not equal to i;

[0044] Screen the benchmark statements and merged statements to remove duplicate data.

[0045] Furthermore, screening the benchmark statements and merged statements to remove duplicate data includes the following sub-steps:

[0046] Selecting a first test number of test sentences from a natural language text database, wherein the test sentences include two sentences with the same meaning but different expressions;

[0047] The test sentences are analyzed for similarity using the Jaccard similarity technique to obtain a similarity estimate for the test sentences; two sentences in the test sentences are marked as test sub-sentences, the number of characters in the test sub-sentences is obtained and the average is calculated to obtain the average number of characters in the test sentences;

[0048] A rectangular coordinate system is established with the average number of words as the X-axis and the similarity estimate as the Y-axis, named the similarity distribution graph, and the average number of words and the similarity valley value of the test sentence are entered into the similarity distribution graph;

[0049] Perform polynomial regression on the similarity distribution graph to obtain the similarity distribution function;

[0050] For any P(m,h), P(m,h) is respectively combined with T(i,1) to T(i,max(j)) to form a group of sentences to be analyzed, where max(j) is the maximum value of j. For a group of sentences to be analyzed, the average number of words in the two sentences to be analyzed is obtained and marked as the number of words to be analyzed;

[0051] Substitute the number of words to be analyzed into the similarity distribution function to calculate the target similarity. At the same time, perform similarity analysis on the sentence to be analyzed using the Jaccard similarity technology to obtain the similarity to be analyzed.

[0052] Compare the similarity to be analyzed with the target similarity. If the similarity to be analyzed is less than the target similarity, output a sentence independence signal; if the similarity to be analyzed is greater than or equal to the target similarity, output a sentence repetition signal;

[0053] If the output statement repeats the signal, then P(m,h) is changed from R m Remove all P(m,h) from the output statement repetition signal and remove them from the corresponding R m After the elimination, R m Marked as E m ;

[0054] Splice E1 into R i After that, start again with 1 and set E m+1 Splice in E m Afterwards, add m+1 and continue splicing until all the splicing is completed, m≠i, and a unified construction file is obtained after all the splicing is completed.

[0055] In a second aspect, the present application provides an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the above method are performed.

[0056] In a third aspect, the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above method are performed.

[0057] The beneficial effects of the present invention are as follows: all construction file data are converted into a unified format to ensure uniform data types, and then the data in the file content is screened to screen out the text part and the table part, and then the text part is matched and aligned by a data alignment algorithm, and the table part and the file table are matched and aligned by a data alignment algorithm, and the aligned construction file data are integrated and merged, and a reference data is screened out as a basis for integration. The advantage of the present invention is that, unlike the existing data alignment technology, since there are duplicate data in the original file itself, the problem of missing content may occur when directly aligning and removing duplicate data. By checking the duplicates between the same source data, the file with the highest degree of duplication is found as the reference file. At this time, it is only necessary to remove the duplicate data of other construction files and merge them, which can ensure that the content of the construction files is not missing, and improve the accuracy and integrity of the construction file data alignment technology;

[0058] The present invention generates a unified data set based on the benchmark data, identifies and removes duplicate data based on the number of words in the sentence, and obtains a similarity distribution function by analyzing the test sentence. The similarity distribution function shows how the similarity between duplicate sentences changes with the number of words. The advantage is that the similarity distribution function can dynamically update the target similarity, so as to more accurately judge whether the sentences to be analyzed are duplicate data based on the number of words in the sentence to be analyzed, thereby improving the accuracy and rationality of the construction document data alignment technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the steps of the method of the present invention;

[0060] Figure 2 is a similarity distribution diagram of the present invention;

[0061] Figure 3 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Example 1, please refer to Figure 1 As shown, the present application provides a construction document data alignment processing method, comprising the following steps:

[0064] Step S1, converting all construction document data into a unified format to ensure uniform data types; Step S1 includes the following sub-steps:

[0065] Step S101, obtaining construction documents, which include text files, table files and picture files;

[0066] Step S102, for a text file, directly extracting text from the text file, and marking the extracted text as file content;

[0067] Step S103, for the table file, directly obtain the entire table, and mark the extracted table as a file table;

[0068] Step S104, for the image file, extract the text in the image file by using image text recognition technology, and mark the extracted text as file content;

[0069] Step S105, the data types of the file content and the file table are both text type;

[0070] In the specific implementation, text files include files with suffixes such as "docx", "doc", "wps", "wpt" and "dot" as text document types, table files include files with suffixes such as "xls", "xlsx", "xlsm", "xlsb" and "csv" as table formats, and image files include files with suffixes such as "png", "jpg", "jpeg", "gif" and "tiff" as picture formats. The text files may contain data in the form of tables and data in the form of pictures. When the data in the form of tables is found, it is the table part in the subsequent analysis, and the data in the form of pictures is first extracted for text and converted into text data. If no text can be extracted from the image file, or the proportion of numbers in the extracted text is greater than 50%, keep the original file and directly merge the construction documents, because image files with a proportion of numbers greater than 50% may be high-precision images such as construction plans and layouts, which need to be understood in conjunction with the image itself, while image files with a low proportion of numbers are usually screenshots of the file or the file is displayed in image format to prevent tampering. At this time, the text can be extracted and converted.

[0071] Step S2, using a data alignment algorithm to match and align construction document data from different data sources; Step S2 includes the following sub-steps:

[0072] Step S201, screening the data in the file content to screen out the text part and the table part;

[0073] Step S201 includes the following sub-steps:

[0074] Step S2011, the file contents are uniformly saved to a word document. In the word document, the table has table attributes and can be directly extracted;

[0075] Step S2012, extracting the table in the file content, marking it as the table part, and marking the remaining file content as the text part;

[0076] Step S2013: The text parts extracted from different construction documents are independent of each other. The text parts are numbered according to the different construction documents. n Indicates that n is a positive integer and n is the serial number of F;

[0077] In the specific implementation, there are three construction documents in this embodiment, which are named as the first file, the second file and the third file respectively. Among them, the first file is a text file, the second file is a table file, and the third file is a picture file. The text parts of the first file and the third file are obtained by extraction and are named as the first text part and the third text part respectively. There is a table part in the first file, which is named as the first table. The file table extracted from the second file is marked as the second table. So far, the first text part, the third text part, the first table and the second table are extracted. The text parts extracted from different construction files are independent of each other, that is, the first text part and the third text part are not related. Through numbering, F1 is the first text part, F2 is the third text part, and 1≤n≤2; since construction files have a certain degree of privacy, the specific content of the construction files is not displayed in this embodiment, and it is only necessary to know the definitions of the text part and the table part;

[0078] Step S202, matching and aligning the text part by a data alignment algorithm, and matching and aligning the table part and the file table by a data alignment algorithm;

[0079] Step S202 includes the following sub-steps:

[0080] Step S2021, setting data features, which include contract number, date, construction location, participating units and person in charge; the date includes the start date and completion date, and the participating units include contractors, subcontractors and supervision units;

[0081] In specific implementation, construction documents usually include multiple documents, including but not limited to construction contracts, construction plans, construction notices and other documents, and these documents usually have inherent features. For example, the construction contract must have a contract number, the construction plan must have a construction date, and the construction notice must have information about the person in charge, and so on. In addition, construction documents often contain information such as the construction location and participating units. This type of information is usually unique, that is, at the same time, the documents at the same construction location must be from the same source, and the construction documents of the same participating units must also be from the same source. Among the same batch of documents that need to be processed, the construction documents from the same source are usually construction documents of the same project. Therefore, data features are set to lock the construction documents of the same project by extracting data features, and then align their data.

[0082] Step S2022, setting a separator, the separator includes symbols "colon", "comma", "period", "semicolon", "space" and "newline";

[0083] Step S2023, searching for data features in the text part, marking them as text features, starting with the text features, searching for the first delimiter and the second delimiter after the text features, marking them as the first delimiter and the second delimiter respectively;

[0084] Step S2024, marking the content between the first partition and the second partition as a text feature value, where the text feature value does not include the first partition and the second partition;

[0085] In a specific implementation, the separators are several commonly used symbols for separating sentences. When searching for data features, for example, searching for the data features of the first text part, it is found that "person in charge" exists in the first text part, and the sentence where the person in charge is specifically "person in charge: Zhang San (line break)", where (line break) is not text content, but represents that a line break is added after the word "three", and a line break operation is performed here. At this time, the text feature is "person in charge", the first separator after the text feature is "colon", and the second separator is "line break", because the first separator is "colon", the second separator is "line break", and the content between the first separator and the second separator is "Zhang San", that is, the text feature value is "Zhang San";

[0086] Step S2025: If the text features are the contract number, construction location and person in charge, directly compare F n If the text feature values ​​are the same, a homology signal is output; if the text feature values ​​are different, a non-homology signal is output;

[0087] Step S2026: If the text feature is a date, compare F nWhether the start date and completion date in the text feature values ​​are the same, if they are the same, output a homology signal; if they are different, output a non-homology signal;

[0088] Step S2027: if the text feature is a participating unit, compare F n Whether the contractors, subcontractors and supervisors in the text feature values ​​are the same, if they are the same, then a homologous signal is output; if they are different, then a non-homologous signal is output;

[0089] Step S2028: If the homologous signal is output, the F n The corresponding construction documents are aligned and marked as homologous data;

[0090] In the specific implementation, when the text features are the contract number, construction location and person in charge, such text features are all single data, while when the text features are dates or participating units, there will be multiple different data, so the corresponding data need to be compared; to better understand the comparison process, this embodiment takes the text feature as a date as an example, assuming that in the text feature value of F1, the start date is July 10, 2024, and the completion date is September 10, 2024, while in the text feature value of F2, the start date is July 10, 2024. The completion date is September 10, 2024. By comparison, it is found that the start date and completion date of F1 and F2 are the same, and a homologous signal is output; and the text features of the first text part and the second text part in this embodiment are actually the person in charge. By comparison, the text feature value of F1 and the text feature value of F2 are both "Zhang San", so a homologous signal is output; the construction files corresponding to F1 and F2 that output the homologous signal are aligned and marked as homologous data, that is, the first file and the third file are homologous data;

[0091] Step S2029, the table parts and file tables extracted from different construction files are independent of each other; the table parts and file tables are collectively referred to as table data, the table header names of the table data are extracted, the table header names of different table data are compared, and the construction files corresponding to the table data with the same header names are marked as homologous data. The integration object of the homologous data is the construction file, so the text part, the table part and the homologous data integrated by the file table are the same;

[0092] In the specific implementation, the header name is the name of the table. Each table has a name to express the meaning of the content in the table. By comparing the header names, it can be confirmed whether the two tables are the same. As long as the same table exists, the corresponding two construction documents must be of the same source; the header name of the first table is obtained as "2024 District A B Street C Construction Consumables Table", and the header name of the second table is obtained as "2024 District A B Street C Construction Consumables Table". By comparing, it is found that the header names of the first and second tables are the same, and a homologous signal is output to integrate the first file and the second file into homologous data, and the first file and the third file are also homologous data. At this time, the homologous data includes the first file, the second file and the third file; in addition, the first table represents the table data extracted from the first file, and there may be multiple tables. The second table represents the table data extracted from the second file, and there may also be multiple tables. As long as there is a table with the same header name among them, it can be identified as homologous data.

[0093] Step S3, integrating and merging the aligned construction document data, and selecting a reference data as the basis for integration; Step S3 includes the following sub-steps:

[0094] Step S301: for any homologous data, the text files, table files and image files in the homologous data are collectively referred to as homologous files, and the homologous files are numbered and represented by symbols R m Represents, where m is a positive integer and m is the sequence number of R;

[0095] Step S302, with any R m As the base file, other R m Compare the duplicate files with the benchmark files, obtain the duplicate rate, and calculate the different R m The sum of the duplicate check rates obtained through analysis is marked as the duplication index;

[0096] Step S302, each R m For the benchmark file, repeat the duplication checking steps to obtain max(m) duplication indexes, where max(m) is the maximum value of m;

[0097] Step S303, finding the maximum value in the repetition index, marking it as the maximum repetition index, and marking the reference file corresponding to the maximum repetition index as the reference data;

[0098] In a specific implementation, the homologous data includes a first file, a second file and a third file, that is, R1 to R3 are respectively the first file, the second file and the third file, 1≤m≤3, and at this time the first file, the second file and the third file all exist in text form, that is, the first file is composed of a first text part and a first table, the second file is composed of a second table, and the third file is composed of a third text part; taking R1 as an example, R2 and R1 are checked for duplicates using existing duplicate checking technology, and a duplicate checking rate of 23% is obtained; R3 and R1 are checked for duplicates, and a duplicate checking rate of 36% is obtained; the calculated repetition index is 59%, and similarly, the calculated repetition indexes of R2 and R3 as benchmark files are 31% and 44% respectively; the maximum repetition index obtained by searching is 59%, corresponding to R1 as the benchmark file, and the benchmark data obtained by processing is R1.

[0099] Step S4, using the benchmark data as a standard, generates a unified data set and removes duplicate data therein; Step S4 includes the following sub-steps:

[0100] Step S401, for the table part and the file table in the homologous data, which are collectively referred to as homologous tables, by comparing the table header names, if there are homologous tables with the same header name, only one copy is retained, and the rest of the same homologous tables are all removed;

[0101] In specific implementation, in this embodiment, R1 includes a first text portion and a first table, R2 includes a second table, and R3 includes a third text portion, and the first table includes 3 tables, namely Table 1.1, Table 1.2 and Table 1.3, and the second table includes 2 tables, namely Table 2.1 and Table 2.2, and Table 1.1, Table 1.2, Table 1.3, Table 2.1 and Table 2.2 are compared in pairs. If there are identical table header names, the redundant ones are removed, and homologous tables in the benchmark data are preferentially retained. For example, in this embodiment, it is found through searching that the table header names of Table 1.1 and Table 2.1 are the same, so Table 2.1 is removed and Table 1.1 is retained. At this time, Table 1.1, Table 1.2, Table 1.3 and Table 2.2 are left in the homologous table, which means that Table 2.1 in the second file has been deleted after removal. At this time, only Table 2.2 exists in the second file, and only Table 2.2 is spliced ​​during subsequent splicing;

[0102] Step S402, for the text part in the homologous data, extract the content between the first line indented to the period in the text part, mark it as a complete sentence, and extract the content between the period and the period, mark it as a complete sentence;

[0103] Step S403: each R mDifferent numbers of complete sentences are obtained by extraction, and the complete sentences extracted from the benchmark data are marked as benchmark sentences, and the remaining complete sentences are marked as merged sentences;

[0104] Step S404: mark m corresponding to the benchmark data as i, number the benchmark statements, and represent them by the symbol T(i,j), where j is a positive integer and (i,j) is the sequence number of T, and T(i,j) represents R i The jth benchmark statement in ;

[0105] Step S405: number the merged statements, represented by the symbol P(m,h), where h is a positive integer and (m,h) is the sequence number of P, and P(m,h) represents R m For the h-th merge statement in P(m,h), m in P(m,h) is not equal to i;

[0106] In the specific implementation, in a paragraph of text, the content between two periods is a complete sentence. If it is the beginning of a paragraph, there will be a first-line indentation format, which also represents the end of the previous sentence and the beginning of the next sentence. Therefore, complete sentences in the same source data are extracted by these two methods; for the benchmark data R1, 34 benchmark sentences are extracted, and 20 merged sentences are extracted in R2; since the benchmark data is R1, m=1 at this time, that is, i=1, and the benchmark sentences T(1,1) to T(1,34) are obtained by numbering, 1≤j≤34, and the merged sentences P(2,1) to P(2,20) are obtained by numbering, 1≤h≤20;

[0107] Step S406, screening the benchmark statements and the merged statements to remove duplicate data therein;

[0108] Step S406 includes the following sub-steps:

[0109] Step S4061, selecting a first test number of test sentences from a natural language text database, wherein the test sentences include two sentences with the same meaning but different expressions;

[0110] Step S4062, performing similarity analysis on the test sentence using the Jaccard similarity technique to obtain a similarity estimate of the test sentence; marking two sentences in the test sentence as test sub-sentences, obtaining the number of characters in the test sub-sentences and calculating the average value to obtain the average number of characters in the test sentence;

[0111] In a specific implementation, a natural language text database includes various sentences in various fields and scenarios, and each sentence has a different way of expression. The first test quantity is set to obtain sufficient sample data to provide a sufficient data basis for the test. There is no specific requirement for the value of the first test quantity, but it shall not be less than 1000 under normal circumstances. The first test quantity in this embodiment is set to 1000, that is, 1000 groups of test sentences are obtained. An arbitrary group of test sentences is used as an example. For example, in the test sentences, the number of words in the first sentence is 34, and the number of words in the second sentence is 38. The average number of words in this group of test sentences is calculated to be 36; Jaccard similarity analysis is performed using the existing Jaccard similarity technology. Jaccard similarity is a metric used to measure the similarity between two sets. It is defined as the size of the intersection of two sets divided by the size of the union. Its value is in the interval [1,2]. The larger the value, the higher the similarity. Due to the large amount of data, the similarity valuation of the test sentences is no longer specifically displayed in this embodiment;

[0112] See also Figure 2 As shown, in step S4063, a plane rectangular coordinate system is established with the average number of words as the X-axis and the similarity estimate as the Y-axis, named as a similarity distribution graph, and the average number of words and the similarity valley value of the test sentence are entered into the similarity distribution graph;

[0113] Step S4064, performing polynomial regression on the similarity distribution graph to obtain a similarity distribution function;

[0114] Step S4065: for any P(m,h), P(m,h) is respectively combined with T(i,1) to T(i,max(j)) to form a group of sentences to be analyzed, where max(j) is the maximum value of j. For a group of sentences to be analyzed, the average number of words of two sentences in the sentence to be analyzed is obtained and marked as the number of words to be analyzed.

[0115] Step S4066, substituting the number of words to be analyzed into the similarity distribution function to calculate the target similarity, and performing similarity analysis on the sentence to be analyzed by using the Jaccard similarity technology to obtain the similarity to be analyzed;

[0116] In the specific implementation, the similarity distribution map is constructed as follows: Figure 2 As shown, the similarity distribution function obtained by regression is Y = -0.00002 × X 2-0.0009×X+1.9282, where Y is the similarity estimate and X is the average number of words. For any P(m,h), for example, P(2,1), P(2,1) and T(1,1) form a group of sentences to be analyzed, and P(2,1) and T(1,2) form a group of sentences to be analyzed. By analogy, for P(2,1), a total of 34 groups of sentences to be analyzed are obtained. For the sentences to be analyzed composed of P(2,1) and T(1,1), the number of words in P(2,1) is 32, and the number of words in T(1,1) is 34. By calculation, the number of words to be analyzed is 33. Substitute X=33 into Y=-0.00002×X 2 -0.0009×X+1.9282, the target similarity is calculated to be 1.877, and the calculation result is rounded to three decimal places; at the same time, the Jaccard similarity technology is used to perform similarity analysis on the sentence to be analyzed, and the similarity to be analyzed is 1.883;

[0117] Step S4067, comparing the similarity to be analyzed with the target similarity, if the similarity to be analyzed is less than the target similarity, outputting a sentence independence signal; if the similarity to be analyzed is greater than or equal to the target similarity, outputting a sentence repetition signal;

[0118] Step S4068: If the output statement repeats the signal, P(m,h) is changed from R m Remove all P(m,h) from the output statement repetition signal and remove them from the corresponding R m After the elimination, R m Marked as E m ;

[0119] Step S4069, splice E1 to R i After that, start again with 1 and set E m+1 Splice in E m After that, add m+1 and continue splicing until all splicing is completed, m≠i, and after all splicing is completed, a unified construction file is obtained;

[0120] In the specific implementation, the similarity to be analyzed is greater than the target similarity through comparison, and a sentence repetition signal is output, and P(2,1) is removed from R2. If P(2,1) has been removed from R2, there is no need to perform similarity analysis on other sentences to be analyzed composed of P(2,1), and P(2,2) is directly analyzed, and so on, until all repeated sentences are removed to obtain E2; since the benchmark data in this embodiment is R1, and m≠i, E1 does not exist, and splicing starts directly from E2, splicing E2 after R1, and E2 is spliced ​​to R1. m After the splicing is completed, the table files in the same source data are spliced ​​after it to obtain a unified construction file.

[0121] Example 2, please refer to Figure 3 As shown, Figure 3 The structural diagram of an electronic device is illustrated, and the electronic device may include: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory stores computer-readable instructions, and the processor can call the instructions in the memory. When the computer-readable instructions are executed by the processor, the steps in a construction file data alignment processing method are executed to achieve the following functions: converting all construction file data into a unified format to ensure the uniformity of data types; using a data alignment algorithm to match and align construction file data from different data sources; integrating and merging the aligned construction file data, and selecting a benchmark data as the basis for integration; using the benchmark data as a standard, generating a unified data set and eliminating duplicate data therein.

[0122] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0123] Embodiment 3, the present application also provides a computer-readable storage medium. The present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above construction document data alignment processing method are executed to achieve the following functions: convert all construction document data into a unified format to ensure the uniformity of data types; use a data alignment algorithm to match and align construction document data from different data sources; integrate and merge the aligned construction document data, and select a benchmark data as the basis for integration; use the benchmark data as the standard to generate a unified data set and eliminate duplicate data therein.

[0124] Through the description of the above implementation methods, the embodiments of the present invention can be provided as methods, systems or computer program products. Based on such an understanding, the above technical solutions can be essentially or partly contributed to the prior art in the form of software products, which can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and include several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0125] In the embodiments provided in the present application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of systems, modules and units can be electrical, mechanical or other forms.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for aligning construction document data, characterized in that: The steps include: Convert all construction document data into a unified format to ensure uniform data types; Use data alignment algorithms to match and align construction document data from different data sources; Integrate and merge the aligned construction document data, and select a benchmark data as the basis for integration; Using the benchmark data as the standard, a unified data set is generated and duplicate data is removed.

2. A construction document data alignment processing method according to claim 1, characterized in that: Converting all construction document data to a uniform format and ensuring uniform naming and data type of data fields includes the following sub-steps: Obtaining construction documents, which include text files, table files, and image files; For text files, directly extract the text in the text files and mark the extracted text as file content; For table files, directly obtain the entire table and mark the extracted table as a file table; For image files, the text in the image files is extracted through image text recognition technology, and the extracted text is marked as file content; The data types of the file content and the file table are both text types.

3. A construction document data alignment processing method according to claim 2, characterized in that: Using the data alignment algorithm, matching and aligning construction document data from different data sources includes the following sub-steps: Filter the data in the file content to select the text part and the table part; The text part is matched and aligned through the data alignment algorithm, and the table part and the file table are matched and aligned through the data alignment algorithm.

4. A method for aligning construction document data according to claim 3, characterized in that: Screening the data in the file content to screen out the text part and the table part includes the following sub-steps: The file contents are uniformly saved to a Word document. In the Word document, the table has table attributes and can be directly extracted; Extract the table in the file content, mark it as the table part, and mark the remaining file content as the text part; The text parts extracted from different construction documents are independent of each other. The text parts are numbered according to the different construction documents. n Indicates that n is a positive integer and n is the sequence number of F.

5. A method for aligning construction document data according to claim 4, characterized in that: Matching and aligning the text part by using the data alignment algorithm, and matching and aligning the table part and the file table by using the data alignment algorithm include the following sub-steps: Setting data features, wherein the data features include contract number, date, construction location, participating units and person in charge; the date includes start date and completion date, and the participating units include contractors, subcontractors and supervision units; Set delimiters, which include the symbols "colon", "comma", "period", "semicolon", "space" and "newline"; Find the data feature in the text part, mark it as the text feature, start with the text feature, find the first delimiter and the second delimiter after the text feature, and mark them as the first delimiter and the second delimiter respectively; Marking the content between the first partition and the second partition as a text feature value, wherein the text feature value does not include the first partition and the second partition; If the text features are the contract number, construction location, and person in charge, then directly compare F n If the text feature values ​​are the same, a homology signal is output; if the text feature values ​​are different, a non-homology signal is output; If the text feature is a date, compare F n Whether the start date and completion date in the text feature values ​​are the same, if they are the same, output a homology signal; if they are different, output a non-homology signal; If the text feature is a participating unit, then compare F n Whether the contractors, subcontractors and supervisors in the text feature values ​​are the same, if they are the same, then a homologous signal is output; if they are different, then a non-homologous signal is output; If the same source signal is output, the F of the same source signal will be output n The corresponding construction documents are aligned and marked as homologous data; The table parts and file tables extracted from the different construction files are independent of each other; the table parts and file tables are collectively referred to as table data, the header names of the table data are extracted, the header names of different table data are compared, and the construction files corresponding to the table data with the same header names are marked as homologous data. The integration object of the homologous data is the construction file, so the homologous data integrated by the text part, the table part and the file table are the same.

6. A method for aligning construction document data according to claim 5, characterized in that: Integrating and merging the aligned construction document data and selecting a benchmark data as the basis for integration includes the following sub-steps: For any homologous data, the text files, table files and image files in the homologous data are collectively referred to as homologous files. The homologous files are numbered and represented by the symbol R. m Represents, where m is a positive integer and m is the sequence number of R; With any R m As the base file, other R m Compare the duplicate files with the benchmark files, obtain the duplicate rate, and calculate the different R m The sum of the duplicate check rates obtained through analysis is marked as the duplication index; Each R m is the benchmark file, and the duplication checking steps are executed cyclically to obtain max(m) duplication indexes, where max(m) is the maximum value of m; Find the maximum value in the repetitiveness index, mark it as the maximum repetitive index, and mark the benchmark file corresponding to the maximum repetitive index as the benchmark data.

7. A method for aligning construction document data according to claim 6, characterized in that: Using the benchmark data as the standard, generating a unified data set and removing duplicate data includes the following sub-steps: For the table parts and file tables in the homologous data, they are collectively referred to as homologous tables. By comparing the table header names, if there are homologous tables with the same header name, only one copy is retained, and the rest of the same homologous tables are all removed; For the text part in the homologous data, extract the content between the first line indented to the period in the text part and mark it as a complete sentence, and extract the content between the period and the period and mark it as a complete sentence; Each R m Different numbers of complete sentences are obtained by extraction, and the complete sentences extracted from the benchmark data are marked as benchmark sentences, and the remaining complete sentences are marked as merged sentences; The m corresponding to the benchmark data is marked as i, and the benchmark statements are numbered and represented by the symbol T(i,j), where j is a positive integer and (i,j) is the sequence number of T, and T(i,j) represents R i The jth benchmark statement in ; The merged statements are numbered and represented by the symbol P(m,h), where h is a positive integer and (m,h) is the sequence number of P. P(m,h) represents R m For the h-th merge statement in P(m,h), m in P(m,h) is not equal to i; Screen the benchmark statements and merged statements to remove duplicate data.

8. A method for aligning construction document data according to claim 7, characterized in that: Screening the benchmark statements and merged statements to remove duplicate data includes the following sub-steps: Selecting a first test number of test sentences from a natural language text database, wherein the test sentences include two sentences with the same meaning but different expressions; The similarity analysis of the test sentences is performed using the Jaccard similarity technique to obtain the similarity estimation of the test sentences; Mark the two sentences in the test sentence as test sub-sentences, obtain the number of words in the test sub-sentences and calculate the average value to obtain the average number of words in the test sentence; A plane rectangular coordinate system is established with the average number of words as the X-axis and the similarity estimate as the Y-axis, named the similarity distribution graph, and the average number of words and the similarity valley value of the test sentence are entered into the similarity distribution graph; Perform polynomial regression on the similarity distribution graph to obtain the similarity distribution function; For any P(m,h), P(m,h) is respectively combined with T(i,1) to T(i,max(j)) to form a group of sentences to be analyzed, where max(j) is the maximum value of j. For a group of sentences to be analyzed, the average number of words in the two sentences to be analyzed is obtained and marked as the number of words to be analyzed; Substitute the number of words to be analyzed into the similarity distribution function to calculate the target similarity. At the same time, perform similarity analysis on the sentence to be analyzed using the Jaccard similarity technology to obtain the similarity to be analyzed. Compare the similarity to be analyzed with the target similarity. If the similarity to be analyzed is less than the target similarity, output a sentence independence signal; if the similarity to be analyzed is greater than or equal to the target similarity, output a sentence repetition signal; If the output statement repeats the signal, then P(m,h) is changed from R m Remove all P(m,h) from the output statement repetition signal and remove them from the corresponding R m After the elimination, R m Marked as E m ; Splice E1 into R i After that, start again with 1 and set E m+1 Splice in E m Afterwards, add m+1 and continue splicing until all the splicing is completed, m≠i, and a unified construction file is obtained after all the splicing is completed.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 8 are executed.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are executed.

Images