A method and system for automatically clustering and quickly identifying attested data

Abstract

The application provides a kind of automatic clustering's certification data fast identification method, this method includes the following steps: response to the template library of establishing XML language's custom label;Based on the corpus extraction of the established template library;The file TF-IDF feature calculation of the file to be imported is carried out based on the extracted corpus, and the features are extracted, and the files with the same features are classified into a class, to complete the automatic classification of file;Further carry out file template matching, if match to template then carry out the matching result in the warehouse operation, otherwise end.Present a kind of based on extension label template library, while relying on corpus to complete the characteristic value calculation of the file to be identified, to complete the automatic classification of file set, rely on automatic classification and characteristic value result cache technology, realize the algorithm and complete system solution of the automatic identification and import of a large number of complex text files, greatly improve the template recognition speed and performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of certificate data analysis, and particularly relates to a fast certificate data identification method and system based on automatic clustering. BACKGROUND

[0002] Certificate data (single data) is the main analysis content in the field of data analysis. The file format and data content format of certificate data are quite different and disorganized.

[0003] Currently, there is no particularly mature and efficient solution for importing certificate data. Because the certificate data is too disorganized and has different formats, for example, the single data of banks and the single data of the five major banks. Due to the differences in platform systems among the five major banks, the exported single data formats of each bank are different. In addition, there are differences in the bank platform of commercial banks and local banks and the differences in the version of the bank system used in different places. The types of exported single data are conservatively estimated to be over ten thousand. During the investigation and evidence collection process, case handling personnel may obtain single data of multiple banks and multiple suspects. When all these data are gathered to case analysis personnel, the analysis personnel usually do not know how to start. The current mainstream methods of importing certificate data include the following methods:

[0004] 1. Import according to the template provided by the system;

[0005] This method is a relatively traditional method, that is, the system only recognizes the template provided by the system itself, and the user needs to convert the original file into the format of the system template and then import it. This method can easily complete the import of a small amount of files with the same format. However, if a large amount of complex single data is faced, the workload is too large.

[0006] 2. The system pre-provisions templates and lets the user select the corresponding template for import;

[0007] This method is equivalent to pre-configuring templates according to the original files obtained by the user. The client needs to select a specific template and upload the corresponding file. This method can quickly complete the import in the case of a small number of files. If the file quantity and file type are too large, the user needs to filter and classify, and then select and upload one template at a time, which is a very large workload.

[0008] 3. According to the title in the data type, filter the table header in the file and perform field matching;

[0009] Such a method can make one-key identification, and no matter what data is imported, the data type and field name are matched with the file content in advance, and when the field quantity is matched to a certain extent, it is considered that the file belongs to this data type; this way is more efficient when corresponding to regular data, but when the file content has a title line or needs to extract key information from some beginning content, the data will be lost, and in addition, it cannot well identify segmented files, that is, the table header appears multiple times in a unified file, and the table header may be different data types or the same data type.

[0010] Therefore, it is very meaningful to provide an automatic clustering and certification data fast identification method and system. SUMMARY

[0011] In order to solve the problems of low import efficiency and large workload of existing certification data, the present application provides an automatic clustering and certification data fast identification method and system to solve the above technical defects.

[0012] In the first aspect, the present application provides an automatic clustering and certification data fast identification method, which comprises the following steps:

[0013] Response to the template library of establishing the custom tag based on XML language;

[0014] Corpus extraction based on the established template library;

[0015] File TF-IDF feature calculation and feature extraction based on the extracted corpus, and files with the same features are classified into a class, and automatic classification of files is completed;

[0016] Further file template matching, if the template is matched, the matching result is stored, otherwise it is ended.

[0017] Preferably, the template library of establishing the custom tag based on XML language comprises: formulating the rules of parsing, identifying and cleaning of this type of file according to the format and content of the original file, and the properties of the template specifically include:

[0018] File attribute, including file format and file parsing program;

[0019] Work sheet, representing a work sheet in the file;

[0020] Segment, representing the content of a data type in the work sheet, and segment definition is to support multi-segment files;

[0021] Standard resource coding, identifying which data type a segment of content belongs to; field mapping configuration, configuring the mapping relationship between original fields and standard fields;

[0022] The cleaning rule configuration configures some cleaning processing rules such as date formatting and standardization conversion.

[0023] Further preferably, the automatic classification of the file includes:

[0024] The batch of files to be identified is automatically classified according to the file feature values, and the files with the same feature values are classified into one category of files. After one template matching of a file in one category of files, the matching results of other files in the same category can be directly reused, so as to reduce the number of template matching of the files and improve the matching performance.

[0025] Further preferably, it further includes: taking the field mapping configuration attribute in the template library as a corpus, calculating the TF-IDF feature of the file, extracting the IDF as a feature character and calculating the MD5, and classifying the files with the same IDF MD5 value into one category of files.

[0026] Further preferably, the file automatic classification process specifically includes:

[0027] A file data set F={f1, f2, … fn} is added, and the data set includes n files to be identified. The sample data subset FS={s1, s2, … sn} of each file is obtained by traversing and parsing and sampling.

[0028] The TF value of the sample data of each file is obtained by traversing FS. TF represents the frequency of each word appearing in the file. The calculation formula of TF is as follows: Where n i,j is the total number of times the word appears in the file, and the denominator is the total number of times of all words in the file. After calculation, the TF value set TFS={tf s1 ,tf s2 ,tf s3 ,…tf sn} of each file is obtained.

[0029] The corpus word set of the template library is calculated. The field mapping configuration in the template library is extracted, and the original field name is extracted as a corpus. LD={t1, t2, t3, … tn}.

[0030] The LD corpus is substituted into the TFS set to calculate the IDF value of each file. The calculation formula of IDF is as follows: Where |D| is the total number of files in the corpus, and |{j: t i ∈d j}| represents the number of files containing the word t i ; After traversing and calculating the TFS set, the IDF set IDFS={idf s1 ,idfs2 idf s3 ,…idf sn};

[0031] Traverse the IDFS set, filter the words with infinite idf value in each element of the set, sort the remaining words according to the ASCII code after filtering, splice the sorted words and calculate the MD5 value, and the MD5 value is used as the classification feature of the file, indicating the file data set of the good classification feature, and is represented as follows: FS={fs1, fs2, …fsn};

[0032] Traverse the FS set, and perform aggregation operation according to the feature value of each element, and identify the files with the same feature value as a class of files, and the aggregated FS set is represented as follows: FSG={{fs1, fs3, …fsm}, {fs2, fs5, …fsn}, {fs10, fs31, …fsx}};

[0033] The automatically classified set FSG is obtained.

[0034] Further preferably, the file template matching process specifically includes:

[0035] Load the classified file set FSG={{fs1, fs3, …fsm}, {fs2, fs5, …fsn}, {fs10, fs31, …fsx}} and traverse the first element of each sub-set, and take out the IDF feature value of the first element of each sub-set to query the cache, if there is a hit result, directly return the result, and if there is no result, proceed to the next step.

[0036] Get all templates in the template library, and filter the templates that may fit the file according to the feature string extracted from the IDF feature value of the file to be matched, and then match the filtered few templates with the file to be matched.

[0037] Match the templates filtered in the previous step with the first element of the classified sub-file set, and whether there is a corresponding matching template or not, the matching result is cached according to the MD5 value of the file IDF feature, and if a file with the same feature value comes in later, the matching result of the file of the same type will be directly used, and the template library does not need to be matched again.

[0038] In the second aspect, the embodiment of the present application also provides a fast identification system for automatically clustered attestation data, which specifically includes:

[0039] A template library module is used to respond to the establishment of a template library of custom tags of XML language;

[0040] A corpus module is used to extract a corpus based on the established template library.

[0041] a feature calculation and extraction module for calculating file TF-IDF features and extracting features of the to-be-imported file based on the extracted corpus;

[0042] an automatic classification module for classifying files with the same features into a category to complete automatic classification of the files;

[0043] a matching module for matching file templates, and if a template is matched, performing a database operation on the matching result, or ending otherwise.

[0044] In a third aspect, an electronic device is provided, including: one or more processors; and a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementation manners of the first aspect.

[0045] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the method described in any of the implementation manners of the first aspect is implemented.

[0046] Compared with the prior art, the beneficial results of the present application are as follows:

[0047] (1) The technical solution of the present application is proposed for the problem of more repeated manual work in the case of importing a large number of complex file data, and a feature value calculation of a to-be-recognized file is completed based on an extended label template library and a corpus, so as to complete automatic classification of a file set, and relying on automatic classification and feature value result caching technology, an algorithm and a complete system solution for automatic recognition and import of a large number of complex text files are realized, which greatly improves the template recognition speed and performance. The system can realize high-performance recognition and matching effect even if it runs on a common PC without relying on parallel and cluster computing.

[0048] (2) As the number of files increases, the time consumption of ordinary matching algorithms increases by several times, but the time consumption of the algorithm of the present application does not increase significantly with the increase of the number of files and classification. In addition, the present application can also obtain high recognition performance on a common machine. For a large number of files with similar data content formats, the recognition performance of the present application is high. If a large number of files have different data formats, the matching performance of the present application is not much different from that of ordinary matching algorithms without using the algorithm of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain principles of the application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0050] Figure 1 is an exemplary device architecture diagram in which an embodiment of the present application can be applied;

[0051] Figure 2 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application;

[0052] Figure 3 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application;

[0053] Figure 4 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application;

[0054] Figure 5 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application;

[0055] Figure 6 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application;

[0056] Figure 7 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application;

[0057] Figure 8 is a flow diagram of a method for fast identification of automatically clustered attestation data according to an embodiment of the present application; DETAILED DESCRIPTION

[0058] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments in which the application can be practiced. In this regard, directional terminology, such as "top," "bottom," "left," "right," "front," "back," etc., is used with reference to the orientation of the Figure(s) being described. Because components of embodiments can be positioned in a number of different orientations, the directional terminology is used for purposes of illustration and is in no way limiting. It is to be understood that other embodiments can be utilized and structural or logical changes can be made without departing from the scope of the present application. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present application is defined by the appended claims.

[0059] It should be appreciated that Figure 1 the number of terminal devices, networks and servers in the system is merely illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs.

[0060] Figure 1 An exemplary system architecture 100 for a method for processing information or an apparatus for processing information to which embodiments of the present application can be applied is shown.

[0061] As shown in Figure 1 the system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0062] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0063] The terminal devices 101, 102, 103 can be various electronic devices with communication functions, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc. The server 105 can be a server providing various services, such as a background information processing server for processing verification request information sent by the terminal devices 101, 102, 103. The background information processing server can analyze and process the received verification request information, and obtain a processing result (e.g. verification success information for indicating that the verification request is a legal request).

[0064] It should be noted that the method for processing information provided by the embodiments of the present application is generally executed by the server 105, and correspondingly, the apparatus for processing information is generally provided in the server 105. In addition, the method for sending information provided by the embodiments of the present application is generally executed by the terminal devices 101, 102, 103, and correspondingly, the apparatus for sending information is generally provided in the terminal devices 101, 102, 103.

[0065] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or multiple software modules, which are not specifically limited here.

[0066] Figure 2 The embodiment of the application shows a kind of automatic clustering attestation data quick identification method, as shown in Figure 2 And Figure 3 The method comprises the following steps:

[0067] S1, response to the establishment of XML language custom tag template library;

[0068] Specifically, the template library in the application refers to the format and content of the original file to formulate the analysis, identification, cleaning and other processing methods of this kind of file;Template is the premise of one-key identification and import, only after the file matches the corresponding template, the program can automatically construct the file analysis method according to the template configuration, compared with the traditional simple table header field template, the definition and composition of the template library in the application are expanded, mainly including the following table:

[0069]

[0070]

[0071] The expanded template attribute can define the specific identification and analysis method of a file, collect general raw files and configure according to each attribute and continuously expand the template library, which forms a certain scale of template library and basically covers all common file analysis methods.

[0072] S2, corpus extraction based on the established template library;

[0073] S3, file TF-IDF feature calculation and feature extraction based on the extracted corpus, and files with the same features are classified into a class, and automatic classification of files is completed;

[0074] Specifically, the function of file classification is to automatically classify a batch of files to be identified and imported according to file feature values, and files with the same feature values are classified into a class of files. Only one file in a class of files needs to be matched with a template once, and other files in the same class can directly reuse the matching result, so as to reduce the number of file template matching and improve the matching performance.

[0075] The application takes the field mapping configuration attribute in the template library as a corpus, calculates the TF-IDF feature (TF-IDF is a statistical method for evaluating the importance of a word for a file or a corpus) of the file, extracts IDF as a feature character and calculates md5, and classifies the files with the same IDF md5 value into a category of files, as shown in Figure 4 The specific steps are as follows:

[0076] S31: Add a file dataset F={f1, f2, … fn}, which contains n files to be identified, traverse and parse and sample the sample data subset FS={s1, s2, … sn} of each file;

[0077] S32: Traverse FS to obtain the TF value of the sample data of each file, and TF represents the frequency of each word appearing in the file. The calculation formula of TF is as follows: Where n i,j is the total number of times the word appears in the file, and the denominator is the total number of times of all words in the file. After calculation, the TF value set TFS={tf s1 ,tf s2 ,tf s3 ,…tf sn} of each file is obtained.

[0078] S33: Calculate the corpus word set of the template library, extract the field mapping configuration in the template library, and extract the original field name as the corpus, LD={t1, t2, t3, … tn};

[0079] S34: Traverse the TFS set and substitute the LD corpus to calculate the IDF value of each file. The calculation formula of IDF is as follows Where |D| is the total number of files in the corpus, and |{j:t i ∈d j} represents the number of files containing the word t i ; After traversing the TFS set, the IDF set IDFS={idf s1 ,idf s2 ,ldf s3 ,…idf sn} of each file is obtained.

[0080] S35: Traverse the IDFS set, filter the words with infinite idf value in each element in the set, sort the remaining words according to the ASSICII code after filtering, splice the sorted words and calculate the MD5 value. The MD5 value is the classification feature of the file, and the file dataset with good classification feature is represented as follows: FS={fs1, fs2, … fsn}.

[0081] S36: Traverse the FS set, perform aggregation operation according to the characteristic value of each element, and identify the files with the same characteristic value as a class of files. The aggregated FS set is represented as follows: FSG = {{fs1, fs3, …fsm}, {fs2, fs5, …fsn}, {fs10, fs31, …fsx}};

[0082] At this point, the automatic classification step of the set FS is completed, and the set FSG after automatic classification is obtained after the above steps.

[0083] S4, further perform file template matching, if the template is matched, the matching result is stored, otherwise, the process is ended.

[0084] Further, after the file automatic classification step is completed, the template recognition and matching step can be performed. The main matching process is as shown in Figure 5

[0085] S41, load the classified file set S32, FSG = {{fs1, fs3, …fsm}, {fs2, fs5, …fsn}, {fs10, fs31, …fsx}}, and traverse the first element of each sub-set. The IDF characteristic value of the first element of each sub-set is extracted to query the cache. If there is a hit result, the result is returned directly. If there is no result, S42 is performed.

[0086] S42, obtain all templates in the template library, and filter the templates that can be matched with the file according to the characteristic string extracted from the IDF characteristic value of the file to be matched. The number of templates filtered in this step is very small compared to the number of templates in the entire template library. Then, the small number of templates filtered are matched with the file to be matched, and the matching speed is greatly improved. When extended to the entire file set, the overall matching performance is also greatly improved.

[0087] S43, use the templates filtered in S42 to match the first element of the classified file set. After matching, whether there is a corresponding matching template or not, the matching result is cached according to the MD5 value of the file IDF characteristic. If a file with the same characteristic value comes in later, the matching result of the file of the same class will be directly used, and the template library does not need to be matched again.

[0088] Specifically, in this embodiment, experiments are performed on a common PC machine with 8G memory and Intel core i5-7400 processor. The experimental results are shown in Figure 6

[0089]

[0090] ​​The experimental conclusion is that, as shown in the experimental chart, with the increase of the number of files, the time consumption of the ordinary matching algorithm increases by several times, and the time consumption of the algorithm of the scheme does not increase significantly with the increase of the number of files and classification. In addition, the scheme can also obtain high recognition performance on ordinary machines. For a large number of files with similar data content format, the recognition performance of the scheme is high. If a large number of files have different data formats, the matching performance of the scheme is not much different from that of the ordinary matching algorithm without using the algorithm.

[0091] The technical scheme of the application aims at the problem of repeated manual work in the case of importing a large number of complex files, and proposes a scheme based on an extended label template library, which simultaneously relies on a corpus to complete the calculation of feature values of files to be identified, so as to complete the automatic classification of the file set. Relying on the automatic classification and feature value result caching technology, the algorithm and complete system solution of automatic recognition and import of a large number of complex text files are realized. The system can realize high-performance recognition and matching effect even when running on an ordinary PC without relying on parallel and cluster computing.

[0092] The application proposes a fast template matching algorithm. When the template library accumulates thousands of templates, the time consumption of a batch of files for recognition will increase by several times. Assuming that the recognition speed of a file is about 5s (the average value of the system test experience), and 500 files are uploaded at the same time, it will take about 50 minutes, and the recognition and matching process will take nearly an hour. The fast matching algorithm proposed by the scheme is based on the TF-IDF feature value and slightly transformed. The template library accumulated by the system is used as a corpus to calculate the feature value of each file according to the file, and the files are automatically classified according to the file feature value. The same type of file is matched only once with the template library. In addition, the template feature value can be used to filter the matching range of the template library, so that the number of matching templates of a type of file is reduced to a minimum. Assuming that the system has 1000 templates, a file needs to match 1000 files before it can be judged which template the file belongs to. After using the TF-IDF feature to filter the template range, a file only needs to match 10 or fewer templates to complete a matching. Through the TF-IDF feature clustering of the template corpus and the template matching range filtering, the template recognition speed and performance are greatly improved.

[0093] In a second aspect, the embodiments of the application also disclose an automatic clustering and fast recognition system of certified data, as shown in Figure 7 The system specifically comprises a template library module 71, a corpus module 72, a feature calculation and extraction module 73, an automatic classification module 74, and a matching module 75.

[0094] In a specific embodiment, the template library module 71 is used to respond to the template library of custom tags established in the XML language; the corpus module 72 is used to extract the corpus based on the established template library; the feature calculation and extraction module 73 is used to calculate the TF-IDF features of the file to be imported based on the extracted corpus and extract the features; the automatic classification module 74 is used to classify files with the same features into one category and complete the automatic classification of files; the matching module 75 is used to match the file template, and if the template is matched, the matching result will be stored in the database, otherwise the process ends.

[0095] Reference below Figure 8 , which shows an electronic device (eg Figure 1 A structural diagram of a computer device 800 (a server or terminal device as shown). Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0096] like Figure 8 As shown, the computer device 800 includes a central processing unit (CPU) 801 and a graphics processing unit (GPU) 802, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 803 or the program loaded from the storage part 809 to the random access memory (RAM) 806. Various programs and data required for the operation of the device 800 are also stored in the RAM 804. The CPU 801, GPU 802, ROM 803 and RAM 804 are connected to each other via a bus 805. An input / output (I / O) interface 806 is also connected to the bus 805.

[0097] The following components are connected to the I / O interface 806: an input section 807 including a keyboard, a mouse, and the like; an output section 808 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 809 including a hard disk and the like; and a communication section 810 including a network interface card such as a LAN card or a modem. The communication section 810 performs communication processing via a network such as the Internet. A drive 811 may also be connected to the I / O interface 806 as needed. A removable medium 812, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 811 as needed, so that a computer program read therefrom can be installed into the storage section 809 as needed.

[0098] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 810, and / or installed from the removable media 812. When the computer program is executed by the central processing unit (CPU) 801 and the graphics processor (GPU) 802, the above-described functions defined in the methods of the present disclosure are performed.

[0099] It should be noted that the computer readable medium described in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium may, for example, be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, device or means, or any combination of the above. More specific examples of the computer readable medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution device, device or means. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transmit a program for use by or in connection with an instruction execution device, device or means. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0100] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0102] The modules described in the embodiments of the present application can be implemented through software or hardware. The modules can be set in a processor.

[0103] As another aspect, the application further provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: respond to the establishment of a template library of custom tags of an XML language; perform corpus extraction based on the established template library; perform file TF-IDF feature calculation and feature extraction on the basis of the extracted corpus, and classify files with the same features into a category, thereby completing automatic classification of the files; and further perform file template matching, and if a template is matched, perform a database operation on the matching result, otherwise, end.

[0104] The above description is merely the preferred embodiments of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the inventive scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) with similar functions to form technical solutions.

Claims

1. A method for automatically clustering and quickly identifying attested data, characterized in that, The method comprises the following steps: Response to the establishment of the template library of the custom tag of XML language; Based on the established template library, the corpus is extracted; The file TF-IDF feature calculation and feature extraction of the to-be-imported file are carried out based on the extracted corpus, and files with the same features are classified into a category, and the automatic classification of the file is completed; The automatic classification of the file comprises: automatically classifying a batch of to-be-recognized imported files according to file feature values, classifying files with the same feature values into a category, and after one template matching of a file in the category, other files in the same category can directly reuse the matching result, so as to reduce the number of file template matching and improve the matching performance; Further, the file template is matched, and if the template is matched, the matching result is stored in the database, otherwise the process is ended; the file template matching process specifically comprises: Load the classified file set, FSG = {{fs1, fs3, …fsm}, {fs2, fs5, …fsn}, {fs10, fs31, …fsx}}, and traverse the first element of each sub-set, take out the IDF feature value of the first element of each sub-set, query the hit cache, and if there is a hit result, directly return the result, otherwise proceed to the next step; Obtain all templates in the template library, and filter the templates suitable for the file according to the feature string extracted from the IDF feature value of the to-be-matched file, and then match the filtered templates with the to-be-matched file; Match the first element of the classified sub-file set with the template filtered in the previous step, and regardless of whether there is a corresponding matching template, the matching result is cached according to the MD5 value of the file IDF feature, and subsequent files with the same feature value will directly use the matching result of the file in the category without matching the template library again.

2. The method of automatically clustering attested data for rapid identification of claim 1, wherein, The template library based on the custom tag of XML language comprises: formulating the rules of parsing, recognizing and cleaning of a certain type of file according to the format and content of the original file, and the attributes of the template specifically comprise: File attribute, including file format and file parsing program; Worksheet, representing a worksheet in the file; Segment, representing the content of a data type in the worksheet, and segment definition is to support multiple segment files; Standard resource coding, indicating which data type a segment of content belongs to; field mapping configuration, configuring the mapping relationship between the original field and the standard field; Cleaning rule configuration, configuring some cleaning processing rules, such as date formatting and standardization conversion.

3. The method of claim 2, wherein, Further comprising: According to the field mapping configuration attribute in the template library as the corpus, the TF-IDF feature of the file is calculated, the IDF is extracted as the feature character and the MD5 is calculated, and the files with the same IDF MD5 value are classified into a category of files.

4. The method of claim 3, wherein, The file automatic classification process specifically comprises: Add a file data set F = {f1, f2, …fn}, which contains n to-be-recognized files, traverse and parse and sample the sample data subset FS = {s1, s2, …sn} of each file; Traverse the FS, get the TF value of each file sample data, TF represents the frequency of each word appearing in the file, the calculation formula of TF is as follows: Where n i,j is the total number of times a certain word appears in the file, the denominator is the total number of times of all words in a file, and the calculated TF value set of each file is TFS={tf s1 ,tf s2 ,tf s3 ,…tf sn}; A corpus word set of the template library is calculated, field mapping configurations in the template library are extracted, and original field names are extracted as a corpus, LD={t1, t2, t3, … tn}; TFS set is traversed, and the LD corpus is substituted into each file to calculate the IDF value of each file. The calculation formula of IDF is as follows: Where |D| is the total number of files in the corpus, |{j:t i ∈d j}| represents the number of files containing the word t i ; after traversing the TFS set, the IDF set IDFS of each file is obtained, IDFS={idf s1 ,idf s2 ,idf s3 ,…idf sn}. The set of IDFS is traversed, words with infinite idf values in each element of the set are filtered, the remaining words are sorted according to the ASSICII code after the filtering, the sorted words are spliced, and an MD5 value is calculated, the MD5 value is used as a classification feature of a file, and a file dataset with a good classification feature is represented as follows: FS={fs1, fs2, … fsn}; The set of FS is traversed, and an aggregation operation is performed according to the feature values of each element, files with the same feature values are identified as a class of files, and the aggregated set of FS is represented as follows: FSG={{fs1, fs3, … fsm}, {fs2, fs5, … fsn}, {fs10, fs31, … fsx}}; An automatically classified set FSG is obtained.

5. An automatically clustered attested data rapid identification system, characterized in that, The system specifically comprises: a template library module configured to establish a template library of custom tags in an XML language; a corpus library module configured to extract a corpus based on the established template library; a feature calculation and extraction module configured to calculate and extract features of a file to be imported based on the extracted corpus; an automatic classification module configured to classify files with the same features into a class to complete automatic classification of the files; a matching module configured to match file templates, and if a template is matched, the matching result is stored, otherwise the process is ended. 6.An electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 4. The program is executed by the processor to implement the method of any one of claims 1 to 4.

Images