An automated file data cleansing method for a file system

Abstract

The present application relates to the technical field of automatic file integration, and particularly relates to an automatic file data cleaning method of a file system, which comprises the following steps: obtaining type identification, requirement information, a plurality of file types and a plurality of to-be-processed contract files included in each file type; obtaining corresponding file complexity according to the data type complexity and data volume of each to-be-processed contract file; obtaining corresponding requirement matching degree according to the file complexity, file name and file content of each to-be-processed contract file and the matching degree of the file content with the requirement information; determining the key files corresponding to each file type according to the requirement matching degree of each to-be-processed contract file; identifying a reference file in the key files, and performing data cleaning on the plurality of to-be-processed contract files based on the reference file. The present application can obtain better data cleaning effect for the plurality of to-be-processed contract files, so as to accurately convert the contract in PDF format into a file in EXCEL format.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic file integration, and in particular to an automatic file data cleaning method of a file system. BACKGROUND

[0002] At present, with the rapid development of digitization, the importance of file system as the core carrier of data storage and management is increasingly prominent. From the massive business files generated by daily operation of enterprises to the huge experimental data files accumulated by scientific research institutions, the data stored in the file system presents an explosive growth trend. With the development of artificial intelligence, various file integration software has emerged in an endless stream. In the process of integrating multiple files, data cleaning (i.e. the process of extracting key data from the original file and converting it into a target format file, while filtering out redundant data in the original file except for key data) is usually required.

[0003] At present, the existing intelligent algorithm cannot accurately identify the key data of the contract file in the PDF format and effectively convert it. Especially in the process of batch conversion of multiple contract files, it cannot identify the data association between multiple contract files, which makes the Excel file converted based on the contract file in PDF format incomplete or have redundant information.

[0004] That is, the data cleaning effect of the intelligent cleaning algorithm provided by the prior art is poor. SUMMARY

[0005] In order to solve the technical problem of poor data cleaning effect of the intelligent cleaning algorithm provided by the prior art, the purpose of the present application is to provide an automatic file data cleaning method of a file system, and the technical solution adopted is as follows:

[0006] In the first aspect, one embodiment of the present application provides an automatic file data cleaning method of a file system, which comprises:

[0007] Obtaining a type identifier, requirement information, a plurality of file types, and a plurality of to-be-processed contract files included in each file type, the type identifier being used to indicate a target file type in the plurality of file types;

[0008] According to the data type complexity and data volume of each to-be-processed contract file, obtaining the file complexity of each to-be-processed contract file;

[0009] According to the first matching degree between the file complexity, the file name and the requirement information of each to-be-processed contract file, and the second matching degree between the file content and the requirement information, obtaining the requirement matching degree of each to-be-processed contract file;

[0010] According to the demand matching degree of each to-be-processed contract file, a key file corresponding to each file type is determined, wherein the key file is a to-be-processed contract file with the highest demand matching degree among a plurality of to-be-processed contract files included in the corresponding file type;

[0011] A reference file is identified from the plurality of key files, wherein the reference file is a key file corresponding to a target file type, and / or a key file corresponding to a file type with the highest type complexity among the plurality of file types, and the type complexity is calculated based on a plurality of file complexities associated with the corresponding file type;

[0012] Data cleaning is performed on the plurality of to-be-processed contract files based on the reference file.

[0013] In one embodiment, the file complexity of each to-be-processed contract file is obtained according to the data type complexity and the data amount of each to-be-processed contract file, comprising:

[0014] A ratio of a first number and a second number corresponding to each to-be-processed contract file is determined as the data type complexity of each to-be-processed contract file, wherein the first number is the number of data types included in the corresponding to-be-processed contract file, and the second number is the total number of data types included in the file type to which the corresponding to-be-processed contract file belongs;

[0015] The product of the data type complexity and the data amount of each to-be-processed contract file is determined as the original complexity of each to-be-processed contract file;

[0016] The original complexity of each to-be-processed contract file is normalized to obtain the file complexity of each to-be-processed contract file.

[0017] In one embodiment, the first matching degree between the file name and the demand information of each to-be-processed contract file is obtained by:

[0018] The file name of each to-be-processed contract file is segmented to obtain the title word group of each to-be-processed contract file, and the demand information is segmented to obtain the demand word group;

[0019] The intersection-union ratio between the title word group and the demand word group of each to-be-processed contract file is determined as the first matching degree between the file name and the demand information of each to-be-processed contract file.

[0020] In one embodiment, the second matching degree between the file content and the demand information of each to-be-processed contract file is obtained by:

[0021] performing word segmentation on the file content of each to-be-processed contract file to obtain a content word group of each to-be-processed contract file, and performing word segmentation on the requirement information to obtain a requirement word group;

[0022] performing semantic extension on the content word group of each to-be-processed contract file to obtain a hypernym word group of each to-be-processed contract file, and performing semantic extension on the requirement word group to obtain a hyponym word group, wherein the first segmented words in the content word group of each to-be-processed contract file and the second segmented words in the corresponding hypernym word group are in one-to-one correspondence, the second segmented words being hypernyms of the corresponding first segmented words, the third segmented words in the requirement word group and the fourth segmented words in the hyponym word group being in one-to-one correspondence, the fourth segmented words being hyponyms of the corresponding third segmented words;

[0023] performing correlation analysis on the hypernym word group and the hyponym word group of each to-be-processed contract file to obtain a second matching degree between the file content and the requirement information of each to-be-processed contract file.

[0024] In an embodiment, the correlation analysis on the hypernym word group and the hyponym word group of each to-be-processed contract file to obtain a second matching degree between the file content and the requirement information of each to-be-processed contract file includes:

[0025] taking the intersection between the hypernym word group and the hyponym word group of each to-be-processed contract file to obtain an intersection word group of each to-be-processed contract file;

[0026] determining the ratio of the number of segmented words included in the intersection word group of each to-be-processed contract file to the number of segmented words included in the hypernym word group as the second matching degree between the file content and the requirement information of each to-be-processed contract file.

[0027] In an embodiment, the requirement matching degree of each to-be-processed contract file is obtained according to the file complexity, the first matching degree between the file name and the requirement information, and the second matching degree between the file content and the requirement information of each to-be-processed contract file, and includes:

[0028] determining the product of the file complexity, the first matching degree between the file name and the requirement information, and the second matching degree between the file content and the requirement information of each to-be-processed contract file as the requirement matching degree of each to-be-processed contract file.

[0029] In an embodiment, the type complexity of each file type is obtained by:

[0030] calculating the mean value of the file complexities of the plurality of to-be-processed contract files included in each file type to obtain a complexity mean value corresponding to each file type;

[0031] According to the complexity average corresponding to each file type, the type complexity of each file type is obtained.

[0032] In one embodiment, according to the complexity average corresponding to each file type, the type complexity of each file type is obtained, including:

[0033] The ratio of the number of the to-be-processed contract files included in each file type to the total number of the to-be-processed contract files included in the plurality of file types is determined as the type coefficient of each file type.

[0034] The product of the complexity average corresponding to each file type and the type coefficient of each file is determined as the type complexity of each file type.

[0035] In one embodiment, the reference file is identified in the plurality of key files, including:

[0036] The target file type is determined as the main type, and the other file types in the plurality of file types except the main type are determined as the secondary types;

[0037] The difference between the type complexity of the main type and the type complexity of each secondary type is determined as the type difference value of each secondary type.

[0038] The type difference values of each secondary type are summed up to obtain a difference sum value.

[0039] In the case where the difference sum value is greater than a preset difference threshold value, the key file corresponding to the target file type is determined as the reference file, wherein the difference threshold value is a negative number.

[0040] In the case where the difference sum value is less than or equal to the difference threshold value, the key file corresponding to the secondary type with the highest type complexity in the plurality of secondary types is determined as the reference file.

[0041] In one embodiment, the obtaining step of the plurality of to-be-processed contract files includes:

[0042] In the plurality of original files, the content of each original file is identified to obtain the original content of each original file, and the original file is a file input into a file system by a user.

[0043] The original content of each original file is preprocessed to obtain the preprocessed content of each original file, wherein the preprocessing includes at least one of missing value processing, abnormal value processing and repeated value processing.

[0044] The file name of each original file and the corresponding preprocessed content are spliced to obtain a plurality of to-be-processed contract files, wherein the file name of the to-be-processed contract file is the file name of the corresponding original file, and the file content of the to-be-processed contract file is the preprocessed content of the corresponding original file.

[0045] In a second aspect, another embodiment of the present application provides an automatic file data cleaning system of a file system, the system comprising:

[0046] The acquisition module is configured to acquire a type identifier, requirement information, a plurality of file types, and a plurality of to-be-processed contract files included in each file type, wherein the type identifier is used to indicate a target file type in the plurality of file types.

[0047] The first processing module is configured to obtain a file complexity of each to-be-processed contract file according to a data type complexity and a data volume of each to-be-processed contract file.

[0048] The second processing module is configured to obtain a requirement matching degree of each to-be-processed contract file according to a first matching degree between the file complexity, the file name, and the requirement information of each to-be-processed contract file, and a second matching degree between the file content and the requirement information.

[0049] The key file determination module is configured to determine a key file corresponding to each file type according to the requirement matching degree of each to-be-processed contract file, wherein the key file is a to-be-processed contract file with the highest requirement matching degree in the plurality of to-be-processed contract files included in the corresponding file type.

[0050] The reference file identification module is configured to identify a reference file in the plurality of key files, wherein the reference file is the key file corresponding to the target file type, and / or the key file corresponding to a file type with the highest type complexity in the plurality of file types, and the type complexity is calculated based on a plurality of file complexities associated with the corresponding file type.

[0051] The data cleaning module is configured to perform data cleaning on the plurality of to-be-processed contract files based on the reference file.

[0052] In a third aspect, another embodiment of the present application further provides an electronic device, which comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the steps of the method of the first aspect.

[0053] In a fourth aspect, another embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method of the first aspect.

[0054] The present application has the following advantages:

[0055] The application first determines the file complexity of each to-be-processed contract file according to the data type complexity and data volume of the to-be-processed contract file, and then determines the requirement matching degree of the to-be-processed contract file according to the first matching degree between the file complexity, the file name and the requirement information of each to-be-processed contract file, the second matching degree between the file content and the requirement information, and the matching degree between the file itself and the requirement information, so as to determine the key file with the highest requirement matching degree in each file type, and then determine the type complexity of each file type according to the file complexity of each to-be-processed contract file, so as to identify the most important key file from the several key files, that is, to select the reference file which matches the user requirement and has strong association with other to-be-processed contract files by comprehensively considering the association between the multiple to-be-processed contract files and the association between the multiple to-be-processed contract files and the requirement information, and finally complete the data cleaning of the multiple to-be-processed contract files according to the reference file, so as to reduce the probability of cleaning the important data errors in the file and reduce the probability of missing the cleaning of the redundant data in the file, so that the multiple to-be-processed contract files can obtain better data cleaning effect, that is, the Excel file converted from the PDF format contract file is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0057] Figure 1 A schematic flow chart of an automatic file data cleaning method of a file system provided by an embodiment of the present application;

[0058] Figure 2 A structural schematic diagram of an automatic file data cleaning system of a file system provided by an embodiment of the present application;

[0059] Figure 3 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined inventive purpose, the specific implementation, structure, features and effects of the automatic file data cleaning method of a file system according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0062] The specific scheme of the automatic file data cleaning method of a file system provided by the present application is specifically described below in combination with the drawings.

[0063] The present application proposes an automatic file data cleaning method of a file system, please refer to Figure 1 which shows a schematic flow chart of the automatic file data cleaning method of a file system according to one embodiment of the present application, which comprises the following steps:

[0064] Step S1, obtaining type identifier, requirement information, a plurality of file types and a plurality of to-be-processed contract files included in each file type.

[0065] Among them, the type identifier is used to indicate the target file type in the plurality of file types.

[0066] In application, the user inputs a plurality of files (i.e. a plurality of to-be-processed contract files included in each file type) that need to be integrated into the file system (also referred to as file integration system), selects the file type (indicated by the aforementioned type identifier) that needs to be generated, and inputs the requirement information (described by text) of the integration process.

[0067] It should be noted that the to-be-processed contract files described in the present application are all PDF format contract files, and the data cleaned files are all EXCEL format files.

[0068] After receiving the plurality of files that need to be integrated input by the user, the file system can determine the aforementioned plurality of file types by classifying the received plurality of files that need to be integrated.

[0069] In application, different to-be-processed contract files can be distinguished based on the identity of the user who inputs the to-be-processed contract files in the file system, for example: the to-be-processed contract files input by user A correspond to file type A, the to-be-processed contract files input by user B correspond to file type B, and so on, and the number of file types is determined by the number of users who can input to-be-processed contract files.

[0070] Alternatively, different to-be-processed contract files can be distinguished based on the contract subject amount associated with the to-be-processed contract files, for example, to-be-processed contract files with a contract subject amount less than or equal to 500,000 RMB correspond to a first file type, to-be-processed contract files with a contract subject amount between 500,000 RMB and 5,000,000 RMB correspond to a second file type, and to-be-processed contract files with a contract subject amount greater than or equal to 5,000,000 RMB correspond to a third file type.

[0071] It should be noted that the distinction of multiple file types can be adaptively set according to actual needs (such as according to the user identity of the input file, the contract subject amount, the industry to which the contract-involved enterprise or organization belongs, etc.), and the present application does not limit this.

[0072] The above type identifier can be a string, such as "file type 1", "file type 2", and "file type 3".

[0073] The above requirement information is used to describe the user's integration requirement / data cleaning requirement for the multiple to-be-processed contract files. For example, the requirement information can be "extract the date data and the corresponding key indicators (such as sales) under the date in all to-be-processed contract files, and sequentially summarize them by time".

[0074] In some embodiments, the step of obtaining the multiple to-be-processed contract files comprises:

[0075] In the multiple original files, the content of each original file is identified to obtain the original content of each original file, and the original file is a file input into the file system by the user;

[0076] The original content of each original file is preprocessed to obtain the preprocessed content of each original file, wherein the preprocessing includes at least one of missing value processing, abnormal value processing, and repeated value processing;

[0077] The file name of each original file and the corresponding preprocessed content are spliced to obtain the multiple to-be-processed contract files, wherein the file name of the to-be-processed contract file is the file name of the corresponding original file, and the file content of the to-be-processed contract file is the preprocessed content of the corresponding original file.

[0078] After the file system receives the multiple original files input by the user, based on the above settings, the completeness and accuracy of the file content of the to-be-processed contract files used in the subsequent process can be guaranteed.

[0079] For example, the above missing value processing can be compensating for missing values based on a fixed preset value, or dynamically compensating for missing values using an interpolation algorithm combined with non-missing data around the missing values.

[0080] The abnormal value processing can be setting the abnormal value as 0 or NULL.

[0081] The repeated value processing can be merging the repeated values.

[0082] In step S2, the file complexity of each to-be-processed contract file is obtained according to the data type complexity and the data amount of each to-be-processed contract file.

[0083] The data type complexity is used to represent the richness of the data types involved in the file content of the to-be-processed contract file. In the present application, the more data types included in a to-be-processed contract file, the higher the possibility of the data type complexity of the to-be-processed contract file.

[0084] For example, the data types involved in the file content of the to-be-processed contract file can include numerical types (such as integer, floating point), text types (such as string, character), date and time types (such as date, time stamp), Boolean types (such as true / false value), and division types (such as enumeration value, label).

[0085] The data amount is the storage space occupied by the to-be-processed contract file, that is, the file size of the to-be-processed contract file. For example, the file size of the to-be-processed contract file is 5MB (data amount).

[0086] In this step, the file complexity of the to-be-processed contract file is comprehensively evaluated from the data type complexity and the data amount of the to-be-processed contract file, which can comprehensively and accurately quantify the complexity of the to-be-processed contract file, that is, the probability of the to-be-processed contract file containing important data required by the user. The higher the file complexity of the to-be-processed contract file, the greater the probability of the to-be-processed contract file containing important data required by the user.

[0087] Specifically, the file complexity of each to-be-processed contract file is obtained according to the data type complexity and the data amount of each to-be-processed contract file, including:

[0088] The ratio of the first number and the corresponding second number of each to-be-processed contract file is determined as the data type complexity of each to-be-processed contract file, wherein the first number is the number of data types included in the corresponding to-be-processed contract file, and the second number is the total number of data types included in the file type to which the corresponding to-be-processed contract file belongs;

[0089] The product of the data type complexity and the data amount of each to-be-processed contract file is determined as the original complexity of each to-be-processed contract file.

[0090] The original complexity of each to-be-processed contract file is normalized to obtain a file complexity of each to-be-processed contract file.

[0091] Exemplarily, the formula of the complexity (file complexity) of the nth to-be-processed contract file under the mth file type can be:

[0092]

[0093] In the above formula, represents the complexity of the nth to-be-processed contract file under the mth file type, represents a normalization function (such as Min-Max normalization or Z-Score standardization), represents the number of data types contained in the nth to-be-processed contract file under the mth file type (i.e., the first number corresponding to the to-be-processed contract file), represents the total number of data types contained in all files under the mth file type (i.e., the second number corresponding to the to-be-processed contract file), represents the data amount of the nth to-be-processed contract file under the mth file type, which can be understood as the aforementioned data type complexity.

[0094] Based on the above settings, the probability that the to-be-processed contract file contains important data required by the user is quantitatively represented by the ratio of the data amount of the data types contained in the to-be-processed contract file to the total number of data types contained in the file type to which the to-be-processed contract file belongs. Instead of taking the total number of data types contained in all to-be-processed contract files as a reference, taking the total number of data types contained in the file type to which the to-be-processed contract file belongs as a reference can accurately represent the richness of the data types involved in the to-be-processed contract file, and thus the data type complexity calculated is more accurate and reliable.

[0095] The normalization measure is to reduce the influence of extreme values (such as excessively large or small data amount) so that the file complexity can accurately represent the richness of the information content contained in the corresponding to-be-processed contract file (i.e., the complexity of the to-be-processed contract file).

[0096] In step S3, the demand matching degree of each to-be-processed contract file is obtained according to the first matching degree between the file complexity, the file name and the demand information of each to-be-processed contract file, and the second matching degree between the file content and the demand information.

[0097] The demand matching degree can be understood as the matching degree between the to-be-processed contract file and the user demand reflected by the demand information.

[0098] In this step, the file complexity of the to-be-processed contract file, the first matching degree between the file name of the to-be-processed contract file and the requirement information, and the second matching degree between the file content of the to-be-processed contract file and the requirement information are comprehensively considered to calculate the requirement matching degree of the to-be-processed contract file, so as to more accurately reflect the matching degree between the to-be-processed contract file and the requirement information.

[0099] In this way, the file name and the file content of the to-be-processed contract file are regarded as different dimensions, and the matching degrees between the two are calculated respectively, so as to avoid the calculation deviation caused by single dimension processing, to comprehensively and accurately analyze the matching degree between the to-be-processed contract file and the requirement information, and to make the finally calculated requirement matching degree more accurate and reliable.

[0100] The first matching degree can be understood as the association degree between the semantics of the file name of the to-be-processed contract file and the semantics of the requirement information, and the second matching degree can be understood as the association degree between the semantics of the file content of the to-be-processed contract file and the semantics of the requirement information.

[0101] For example, the process of obtaining the first matching degree can be as follows:

[0102] The file name of the to-be-processed contract file and the requirement information are preprocessed respectively to remove the file extension included in the file name and the punctuation and stop words included in the requirement information.

[0103] The preprocessed file name and the requirement information are respectively encoded into vectors of fixed dimensions using a pre-trained semantic model (such as Sentence-BERT, Universal Sentence Encoder, etc.), and the cosine similarity or dot product similarity of the two vectors is calculated and taken as the first matching degree.

[0104] For example, the process of obtaining the second matching degree can be as follows:

[0105] The file content of the to-be-processed contract file and the requirement information are preprocessed respectively to clean up the punctuation, stop words and other contents included in the file content of the to-be-processed contract file and the requirement information.

[0106] The preprocessed file content and the requirement information are respectively subjected to keyword extraction (such as by TF-IDF algorithm) to obtain a content keyword set and a requirement keyword set respectively.

[0107] Then, the intersection-union ratio of the content keyword set and the requirement keyword set is calculated and determined as the second matching degree.

[0108] Step S4, determining a key file corresponding to each file type according to the demand matching degree of each to-be-processed contract file.

[0109] The key file is a to-be-processed contract file with the highest demand matching degree among a plurality of to-be-processed contract files of the corresponding file type.

[0110] Step S5, identifying a reference file from the plurality of key files.

[0111] The reference file is a key file corresponding to the target file type and / or a key file corresponding to a file type with the highest type complexity among the plurality of file types, and the type complexity is calculated based on a plurality of file complexities associated with the corresponding file type.

[0112] The plurality of key files correspond to the plurality of file types one-to-one.

[0113] In an example, the mean of the plurality of file complexities of the plurality of to-be-processed contract files included in each file type can be determined as the type complexity of each file type.

[0114] The type complexity is used to quantitatively represent the complexity of the plurality of to-be-processed contract files included in the corresponding file type, i.e., the comprehensive probability that the plurality of to-be-processed contract files included in the corresponding file type contain important data required by the user.

[0115] By comparing the difference between the type complexity of the target file type and the type complexity of other file types, a more reliable reference file can be determined.

[0116] For example, the number of file types other than the target file type among the plurality of file types can be counted and defined as a first number, and the number of file types other than the target file type among the plurality of file types with a type complexity higher than that of the target file type can be counted and defined as a second number.

[0117] The ratio of the second number to the first number is calculated, and the ratio is compared with a set proportion threshold value (such as 0.3 or 0.35). If the ratio is greater than the set proportion threshold value, the key file corresponding to the file type with the highest type complexity among the plurality of file types other than the target file type is selected as the reference file. If the ratio is less than or equal to the set proportion threshold value, the key file corresponding to the target file type is selected as the reference file.

[0118] Step S6, performing data cleaning on the plurality of to-be-processed contract files based on the reference file.

[0119] Exemplarily, the process of data cleaning of the plurality of to-be-processed contract files based on the benchmark file can be as follows:

[0120] The benchmark file is first parsed to obtain the field specifications (such as field name, data type, constraint condition (such as non-empty, uniqueness, value range, etc.)) and association relationship (such as foreign key constraint) of the benchmark file, and field description and annotation information are extracted to form a field mapping table as a data cleaning standard;

[0121] Subsequently, the to-be-processed contract files other than the benchmark file are read, and each to-be-processed contract file is uniformly converted into a target intermediate format (such as DataFrame) according to the field mapping table, and format difference problems (such as date field unified to "YYYY-MM-DD" format, numerical value field removing thousand separator, etc.) are automatically processed in the process;

[0122] After the format conversion is completed, the plurality of to-be-processed contract files are converted into EXCEL format cleaned files according to the constraint rules (referring to field specifications and association relationship) of the benchmark file, and each time the data cleaning operation is executed, log records, abnormal data and data cleaning details are generated.

[0123] Among them, during the execution of the data cleaning operation process, for repeated values, de-duplication processing is performed according to the primary key field, and the first occurrence record is retained or the latest record is retained according to the time stamp; for missing values, the default value configured by the benchmark file is filled (such as filling 0 for numerical value type, filling "unknown" for text type) or supplemented according to the association field through other data table (such as associating customer information through order ID); for abnormal values, automatic correction or marking according to the preset range (such as age 0-120 years old); for association verification, the foreign key reference integrity is checked (such as the department ID in the employee table must exist in the department table).

[0124] The application first determines the file complexity of each to-be-processed contract file according to the data type complexity and data volume of the to-be-processed contract file, and then determines the demand matching degree of the to-be-processed contract file from the adaptation degree between the file itself and the demand information and other aspects according to the first matching degree between the file name and the demand information of each to-be-processed contract file and the second matching degree between the file content and the demand information, so as to determine the key file with the highest demand matching degree in each file type, and then determine the type complexity of each file type according to the file complexity of each to-be-processed contract file, so as to identify the most important key file from the several key files, that is, by comprehensively considering the relevance between the multiple to-be-processed contract files and the relevance between the multiple to-be-processed contract files and the demand information, the reference file that matches the user demand and has strong relevance with other to-be-processed contract files is selected, and finally the data cleaning of the multiple to-be-processed contract files is completed according to the reference file, which can reduce the probability of cleaning the important data error in the file and reduce the probability of missing the cleaning of the redundant data in the file, so that the multiple to-be-processed contract files can obtain better data cleaning effect.

[0125] In one embodiment, the first matching degree between the file name and the demand information of each to-be-processed contract file is obtained by:

[0126] performing word segmentation processing on the file name of each to-be-processed contract file to obtain the title word group of each to-be-processed contract file, and performing word segmentation processing on the demand information to obtain the demand word group;

[0127] determining the intersection-union ratio between the title word group and the demand word group of each to-be-processed contract file as the first matching degree between the file name and the demand information of each to-be-processed contract file.

[0128] Exemplarily, the above word segmentation processing can be completed by using the Jieba word segmentation tool.

[0129] The intersection-union ratio between the title word group and the demand word group of each to-be-processed contract file can also be understood as the Jaccard correlation coefficient between the title word group and the demand word group of each to-be-processed contract file.

[0130] For example, if the title word group of a to-be-processed contract file is [a1, a2, a3, a4], and the demand word group is [a2, a3, a5], the intersection between the title word group and the demand word group is [a2, a3], the union is [a1, a2, a3, a4, a5], and the intersection-union ratio is 0.4, that is, the first matching degree between the to-be-processed contract file and the demand information is 0.4.

[0131] In the embodiment, the first matching degree between the file name of each to-be-processed contract file and the requirement information is determined by content segmentation and calculation of the intersection and union ratio between different word groups obtained by segmentation, so as to improve the data cleaning efficiency of the method.

[0132] In one embodiment, the second matching degree between the file content of each to-be-processed contract file and the requirement information is obtained by:

[0133] segmenting the file content of each to-be-processed contract file to obtain content word groups of each to-be-processed contract file, and segmenting the requirement information to obtain requirement word groups;

[0134] performing semantic extension on the content word groups of each to-be-processed contract file to obtain superordinate word groups of each to-be-processed contract file, and performing semantic extension on the requirement word groups to obtain subordinate word groups, wherein each first segmented word in the content word groups of each to-be-processed contract file corresponds to a second segmented word in the corresponding superordinate word group, the second segmented word being a superordinate word of the corresponding first segmented word, and each third segmented word in the requirement word groups corresponds to a fourth segmented word in the corresponding subordinate word group, the fourth segmented word being a subordinate word of the corresponding third segmented word;

[0135] performing correlation analysis on the superordinate word groups and the subordinate word groups of each to-be-processed contract file to obtain the second matching degree between the file content of each to-be-processed contract file and the requirement information.

[0136] In the present application, semantic extension is used to find the superordinate / subordinate words of the input segmented words in the word library according to the pre-established semantic mapping. The superordinate word generally represents a more extensive and general concept, which includes the concept represented by the corresponding subordinate word, for example, "animal" is the superordinate word of "dog", and "fruit" is the superordinate word of "apple". The subordinate word generally represents a more specific and particular concept, which belongs to a subclass of the concept represented by the corresponding superordinate word, for example, "rose" is the subordinate word of "flower", and "golden retriever" is the subordinate word of "dog". The foregoing semantic mapping is used to describe the hierarchical relationship between the words in the word library.

[0137] In the embodiment, the first matching degree between the file name of each to-be-processed contract file and the requirement information is determined by content segmentation and calculation of the intersection and union ratio between different word groups obtained by segmentation, so as to improve the data cleaning efficiency of the method.

[0138] Similarly, the one-to-one correspondence between the third segmented words included in the requirement word groups and the fourth segmented words included in the subordinate word groups is that any one of the third segmented words included in the requirement word groups can find a corresponding plurality of subordinate words in the corresponding subordinate word groups.

[0139] In the embodiment, the hypernyms of each of the content word groups of the to-be-processed contract file are obtained to form the hypernym word groups of the to-be-processed contract file, the hyponyms of each of the demand word groups of the demand information are obtained to form the hyponym word groups, and the second matching degree between the file content of the to-be-processed contract file and the demand information is determined by the correlation analysis of the two, so that the semantic relationship between the hypernyms and the hyponyms is utilized to conveniently and accurately quantify the matching degree between the file content of the to-be-processed contract file and the demand information.

[0140] In the above setting, the hyponyms of each of the demand word groups of the demand information are obtained, so that the content of the demand information is reasonably expanded to fully reflect the semantic relevance between the demand information and the file content of the to-be-processed contract file in cooperation with the hyponym word groups of the to-be-processed contract file, so that the second matching degree obtained is more accurate and reliable.

[0141] The correlation analysis of each of the hypernym word groups and the hyponym word groups of the to-be-processed contract file obtains the second matching degree between the file content of each of the to-be-processed contract file and the demand information, including:

[0142] The intersection of each of the hypernym word groups and the hyponym word groups of the to-be-processed contract file is obtained to obtain the intersection word groups of each of the to-be-processed contract file.

[0143] The ratio of the number of the included word segments of each of the intersection word groups of the to-be-processed contract file to the number of the included word segments of the hypernym word groups is determined as the second matching degree between the file content of each of the to-be-processed contract file and the demand information.

[0144] For example, the word segment array (i.e., the aforementioned content word group) of the nth to-be-processed contract file under the mth file type is denoted as word segment array n. The hypernyms of each of the word segments in the word segment array n are obtained by using WordNet to form the hypernym word groups of the nth to-be-processed contract file under the mth file type. If there are R word segments in the word segment array n, the hypernym word groups of the nth to-be-processed contract file under the mth file type will include R' hypernyms, and R' is less than or equal to R.

[0145] It is determined whether the R' hypernyms appear in the hyponym word groups corresponding to the demand information. If the R' hypernyms appear, the word segment correlation degree between the file content of the nth to-be-processed contract file under the mth file type and the demand information is processed by +1, and then the word segment correlation degree between the content of the nth to-be-processed contract file under the mth file type and the demand information is obtained. .

[0146] The mathematical formula of the matching degree (i.e., the second matching degree) between the file content and the requirement information of the nth to-be-processed contract file under the mth file type can be expressed as:

[0147]

[0148] In the formula, m represents the mth file type, n represents the nth to-be-processed contract file under the mth file type, and R represents the number of the segmented words included in the segmented word array n. The matching degree between the file content and the requirement information of the nth to-be-processed contract file under the mth file type is represented by R. The segmented word correlation degree between the file content and the requirement information of the nth to-be-processed contract file under the mth file type is represented by R, and R represents the number of the segmented words included in the segmented word array n.

[0149] In one embodiment, the requirement matching degree of each to-be-processed contract file is obtained according to the file complexity, the first matching degree between the file name and the requirement information, and the second matching degree between the file content and the requirement information of each to-be-processed contract file, and includes:

[0150] The product of the file complexity, the first matching degree between the file name and the requirement information, and the second matching degree between the file content and the requirement information of each to-be-processed contract file is determined as the requirement matching degree of each to-be-processed contract file.

[0151] For example, if the requirement matching degree of the nth to-be-processed contract file under the mth file type is set as R, the mathematical formula of R can be expressed as:

[0152]

[0153] In the formula, m represents the mth file type, n represents the nth to-be-processed contract file under the mth file type, and R represents the number of the segmented words included in the segmented word array n. The second matching degree between the file content and the requirement information of the nth to-be-processed contract file under the mth file type is represented by R. The file complexity of the nth to-be-processed contract file under the mth file type is represented by C. The first matching degree between the file name and the requirement information of the nth to-be-processed contract file under the mth file type is represented by R.

[0154] In one embodiment, the type complexity of each file type is obtained by:

[0155] The average value of the file complexities of the plurality of to-be-processed contract files included in each file type is calculated to obtain the complexity average value corresponding to each file type.

[0156] The type complexity of each file type is obtained according to the complexity average value corresponding to each file type.

[0157] ​The type complexity of each file type is obtained according to the complexity average corresponding to each file type, including:

[0158] The type coefficient of each file type is determined as the ratio of the number of the to-be-processed contract files included in each file type to the total number of the to-be-processed contract files included in the plurality of file types.

[0159] The type complexity of each file type is determined as the product of the complexity average corresponding to each file type and the type coefficient of each file.

[0160] For example, if it is set that The type complexity of the mth file type in the plurality of file types is represented as The calculation formula of the type complexity can be represented as:

[0161]

[0162] wherein, The number of the to-be-processed contract files included in the mth file type is represented as The total number of the to-be-processed contract files included in the plurality of file types is represented as M is the number of the plurality of file types, The file complexity of the nth to-be-processed contract file under the mth file type is represented as The type coefficient of the mth file type can be understood as The average of the plurality of file complexities corresponding to the mth file type can be understood as

[0163] In this embodiment, the importance of each file type in the plurality of file types is represented based on the ratio of the number of the to-be-processed contract files included in each file type to the total number of the to-be-processed contract files, and the importance of the plurality of to-be-processed contract files included in each file type is represented based on the average of the plurality of file complexities of the plurality of to-be-processed contract files, so as to comprehensively and accurately evaluate the importance (i.e., the type complexity) of each file type by comprehensively analyzing the importance of each file type in the horizontal direction (among the plurality of file types) and the vertical direction (the plurality of to-be-processed contract files), so that the determined type complexity is more accurate.

[0164] In one embodiment, the reference file is identified in the plurality of key files, including:

[0165] The target file type is determined as the main type, and the other file types in the plurality of file types except the main type are determined as the secondary types;

[0166] The difference between the type complexity of the main type and the type complexity of each sub-type is determined as a type difference value of each sub-type;

[0167] The type difference values of each sub-type are summed up to obtain a difference sum value;

[0168] In the case where the difference sum value is greater than a preset difference threshold value, the key file corresponding to the target file type is determined as the reference file, wherein the difference threshold value is a negative number;

[0169] In the case where the difference sum value is less than or equal to the difference threshold value, the key file corresponding to the sub-type with the highest type complexity in the plurality of sub-types is determined as the reference file.

[0170] The difference sum value is used to represent the degree of difference between the type complexity of the target file type and the type complexity of other file types.

[0171] For example, if the difference sum value is set as D, the calculation formula of D can be represented as:

[0172]

[0173] wherein, represents the type complexity of the main type, represents the type complexity of the jth sub-type in the M file types, and M is the number of the plurality of file types. In this example, the difference threshold value can be set as -0.25.

[0174] In the case where the difference sum value is greater than the preset difference threshold value (i.e. the case where the difference sum value is positive or the case where the difference sum value is a smaller negative number), it means that in the plurality of file types, the file type with the highest complexity is the main type (or the complexity of the main type is in the top position), that is, the main type is not only the file type specified by the user, but also the content richness of the main type including the plurality of to-be-processed contract files (the number of to-be-processed contract files, the data amount of each to-be-processed contract file, and the number of data types of each to-be-processed contract file) is in the top position or in the top position among the plurality of file types. Therefore, the key file corresponding to the main type is determined as the reference file, which can more effectively guide the data cleaning work of the remaining plurality of to-be-processed contract files.

[0175] In the case that the difference sum is less than or equal to the difference threshold (i.e. the difference sum is a larger negative number), it means that the main type, although being the file type specified by the user, has a relatively low content richness of the plurality of to-be-processed contract files included in the main type, and thus, if the key file corresponding to the main type is mechanically determined as the reference file, a large error is likely to occur in the data cleaning work of the remaining plurality of to-be-processed contract files. In this case, the key file corresponding to the secondary type with the highest type complexity in the plurality of secondary types is selected as the reference file, which can effectively overcome the interference problem caused by the inappropriate file type specified by the user, so that the data cleaning work of the remaining plurality of to-be-processed contract files can be better performed, that is, the plurality of PDF format to-be-processed contract files can be more accurately converted into EXCEL file format.

[0176] The present application provides an automatic file data cleaning system of a file system, please refer to Figure 2 which shows a structural schematic diagram of an automatic file data cleaning system 200 of a file system according to an embodiment of the present application, and the system comprises:

[0177] The acquisition module 201 is configured to acquire a type identifier, requirement information, a plurality of file types, and a plurality of to-be-processed contract files included in each file type, and the type identifier is used to indicate a target file type in the plurality of file types.

[0178] The first processing module 202 is configured to obtain a file complexity of each to-be-processed contract file according to a data type complexity and a data amount of each to-be-processed contract file.

[0179] The second processing module 203 is configured to obtain a requirement matching degree of each to-be-processed contract file according to a first matching degree between the file complexity of each to-be-processed contract file, a file name and the requirement information, and a second matching degree between a file content and the requirement information.

[0180] The key file determination module 204 is configured to determine a key file corresponding to each file type according to the requirement matching degree of each to-be-processed contract file, wherein the key file is a to-be-processed contract file with the highest requirement matching degree in the plurality of to-be-processed contract files included in the corresponding file type.

[0181] The reference file identification module 205 is configured to identify a reference file in the plurality of key files, wherein the reference file is the key file corresponding to the target file type, and / or the key file corresponding to the file type with the highest type complexity in the plurality of file types, and the type complexity is calculated based on the plurality of file complexities associated with the corresponding file type.

[0182] The data cleaning module 206 is configured to clean data of the plurality of to-be-processed contract files based on the benchmark file.

[0183] It should be noted that the system provided in the above embodiments is only used as an example for the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above-described functions. In addition, the automatic file data cleaning system and the automatic file data cleaning method of the file system provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0184] The embodiment of the application further provides an electronic device. Please refer to Figure 3 The electronic device can include a processor 301, a memory 302, and a program 3021 stored in the memory 302 and executable on the processor 301.

[0185] The program 3021, when executed by the processor 301, can implement Figure 1 Any step in the corresponding method embodiment and achieve the same beneficial effects, which will not be repeated here.

[0186] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by program instructions related to hardware, and the program can be stored in a readable medium.

[0187] The embodiment of the application further provides a readable storage medium, and the readable storage medium stores a computer program. The computer program is executed by the processor to implement any step in the above Figure 1 corresponding method embodiments, and can achieve the same technical effects. To avoid repetition, it will not be repeated here.

[0188] The computer readable storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0189] The computer readable signal medium can include a computer readable program code in a baseband or propagated as a carrier wave in a propagation medium. Such a propagated signal can take a wide variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium can be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0190] The program code contained on the storage medium can be transmitted by any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0191] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In an embodiment of the application, the remote computer can be a server or another desktop computer.

[0192] The embodiment of the present application further provides a computer program product, which, when running on a computer, enables the computer to execute the above related steps to realize the automatic file data cleaning method of the file system provided by the above embodiment.

[0193] It should be noted that the above-mentioned embodiment sequence is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0194] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment mainly explains the difference from other embodiments.

Claims

1. An automated file data cleaning method for a file system, characterized in that, The methods include: Obtain the type identifier, requirement information, multiple file types, and multiple pending contract files included in each file type. The type identifier is used to indicate the target file type among multiple file types. The file complexity of each contract file to be processed is obtained based on the data type complexity and data volume of each contract file to be processed. The requirement matching degree of each contract file to be processed is obtained based on its file complexity, file name and first matching degree between the requirement information, and file content and second matching degree between the requirement information. Based on the requirement matching degree of each pending contract file, the key file corresponding to each file type is determined. Among them, the key file is the pending contract file with the highest requirement matching degree among the multiple pending contract files included in the corresponding file type. Identify a benchmark file among multiple key files, where the benchmark file is: the key file corresponding to the target file type, and / or the key file corresponding to the file type with the highest type complexity among multiple file types, and the type complexity is calculated based on the complexity of multiple files associated with the corresponding file type; Data cleaning is performed on multiple contract files to be processed based on the baseline file; The steps for obtaining the initial match between the filename of each pending contract file and the requirement information include: The filename of each contract file to be processed is segmented into words to obtain the title phrase of each contract file to be processed; and the requirement information is segmented into words to obtain the requirement phrase. The intersection-union ratio between the title phrase and the requirement phrase of each contract file to be processed is determined as the first matching degree between the file name and the requirement information of each contract file to be processed. The steps for obtaining the second degree of matching between the content of each pending contract document and the requirement information include: The content of each contract file to be processed is segmented into words to obtain the content phrases of each contract file to be processed; and the requirement information is segmented into words to obtain the requirement phrases. Semantic expansion is performed on the content phrases of each contract document to be processed to obtain the superordinate phrases of each contract document to be processed; and semantic expansion is performed on the demand phrases to obtain the subordinate phrases. Among them, multiple first words in the content phrases of each contract document to be processed correspond one-to-one with multiple second words in the corresponding superordinate phrases, and the second words are the superordinate words of the corresponding first words. Multiple third words included in the demand phrases correspond one-to-one with multiple fourth words included in the subordinate phrases, and the fourth words are the subordinate words of the corresponding third words. For each contract document to be processed, perform correlation analysis on the hypernyms and hyponyms to obtain the second degree of matching between the document content and the requirement information of each contract document to be processed.

2. The automated file data cleaning method for a file system according to claim 1, characterized in that, Based on the data type complexity and data volume of each contract file to be processed, the file complexity of each contract file to be processed is obtained, including: The ratio of the first number to the second number corresponding to each contract file to be processed is determined as the data type complexity of each contract file to be processed, where the first number is the number of data types included in the corresponding contract file to be processed, and the second number is the total number of data types included in the file type to which the corresponding contract file to be processed belongs. The original complexity of each contract file to be processed is determined by multiplying the data type complexity and the data volume of each contract file to be processed. The original complexity of each contract file to be processed is normalized to obtain the file complexity of each contract file to be processed.

3. The automated file data cleaning method for a file system according to claim 1, characterized in that, For each contract document to be processed, perform correlation analysis on its hypernyms and hyponyms to obtain the second degree of matching between the document content and the requirement information of each contract document, including: Take the intersection between the hypernym and hyponym of each contract document to be processed to obtain the intersection phrase of each contract document to be processed. The ratio of the number of words included in the intersection phrase of each contract document to the number of words included in the hypernym phrase is determined as the second degree of matching between the document content and the requirement information of each contract document.

4. The automated file data cleaning method for a file system according to claim 1, characterized in that, Based on the file complexity, the first degree of matching between the file name and the requirement information, and the second degree of matching between the file content and the requirement information, the requirement matching degree of each contract file to be processed is obtained, including: The requirement matching degree of each contract file to be processed is determined by multiplying the file complexity, the first matching degree between the file name and the requirement information, and the second matching degree between the file content and the requirement information.

5. The automated file data cleaning method for a file system according to claim 1, characterized in that, The steps to obtain the type complexity of each file type include: Calculate the average file complexity of multiple contract files to be processed for each file type, and obtain the average complexity for each file type. The type complexity of each file type is obtained based on the average complexity of each file type.

6. The automated file data cleaning method for a file system according to claim 5, characterized in that, Based on the average complexity of each file type, the type complexity of each file type is obtained, including: The ratio of the number of pending contract files included in each file type to the total number of pending contract files included in multiple file types is determined as the type coefficient for each file type. The type complexity of each file type is determined by multiplying the average complexity of each file type by the type coefficient of each file type.

7. The automated file data cleaning method for a file system according to claim 1, characterized in that, Identify the benchmark file among several key files, including: The target file type is determined as the primary type, and other file types besides the primary type are determined as secondary types. The difference between the type complexity of the main type and the type complexity of each subtype is determined as the type difference value of each subtype; The sum of the differences for each subtype is calculated to obtain the sum of the differences. If the difference value is greater than the preset difference threshold, the key file corresponding to the target file type is determined as the baseline file, where the difference threshold is a negative number; If the difference sum is less than or equal to the difference threshold, the key file corresponding to the subtype with the highest type complexity among multiple subtypes is determined as the baseline file.

8. The automated file data cleaning method for a file system according to claim 1, characterized in that, The steps for obtaining multiple pending contract documents include: In multiple original files, content recognition is performed on each original file to obtain the original content of each original file. The original files are files in the user input file system. The original content of each original file is preprocessed to obtain the preprocessed content of each original file. The preprocessing includes at least one of missing value handling, outlier handling, and duplicate value handling. The filename of each original file and its corresponding preprocessed content are concatenated to obtain multiple contract files to be processed. The filename of the contract file to be processed is the filename of the corresponding original file, and the content of the contract file to be processed is the preprocessed content of the corresponding original file.

Images