Method and system for automatically matching and classifying titles of digitized personnel files
By using the TFIDF algorithm and cosine similarity algorithm to automatically process personnel file titles, the problem of low efficiency and large deviation in traditional manual classification is solved, and efficient and unified automatic file classification is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-31
- Publication Date
- 2026-03-03
AI Technical Summary
Traditional digital archive classification relies on manual operation, which is inefficient and prone to classification errors, resulting in inconsistent classification standards and a significant waste of human, material, and financial resources.
By employing the TFIDF and cosine similarity algorithms, and establishing a template library for archive categories, the system extracts title information using word segmentation encoding and OCR technology, calculates word segmentation weights and similarities, and achieves automatic classification.
It improves the efficiency of file classification, reduces human error, saves human, material and financial resources, and ensures the consistency of classification standards.
Smart Images
Figure CN115617985B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of archives management technology, and specifically relates to a method and system for automatic matching and classification of digital personnel file titles. Background Technology
[0002] Digital archives are playing an increasingly important role in archive review and management due to their convenience, ease of storage, management, and queryability. A crucial aspect of personnel file review is the classification and archiving of these files. Original personnel files consist of historical and comprehensive materials reflecting the cadre's situation. According to the "Regulations on Cadre Archives Work," original personnel file materials can be divided into ten categories, including resume materials, autobiographies and ideological materials, performance evaluation materials, etc. Due to the complexity and diversity of the formats used in various personnel files, achieving automated classification and archiving is not only the core of digital archive management but also a significant challenge.
[0003] Traditional classification of digital archives largely relies on manual work by archivists. This process is not only inefficient, but even trained professionals are prone to errors due to their subjective biases and limited knowledge, resulting in inconsistent classification standards. Furthermore, manual classification of digital archives is extremely costly in terms of manpower, resources, and finances. Because the formats of archival documents vary considerably, it is currently difficult to define a standardized template for the entire document. Summary of the Invention
[0004] To this end, the present invention provides a method and system for automatic matching and classification of digital personnel file titles, which uses the TFIDF algorithm and cosine similarity algorithm to achieve text matching and automatic classification and archiving of digital personnel file titles, thereby improving the efficiency of file management.
[0005] According to the design scheme provided by this invention, a method for automatic matching and classification of digital personnel file titles is provided, comprising the following:
[0006] Based on the mapping relationship between the title of each personnel file and its corresponding file category, a file category template library is established;
[0007] The personnel file titles in the file category template library are vectorized, and a corpus is created by word segmentation encoding. The TFIDF model is used to store the word segmentation weight value of each title.
[0008] Extract the title information of the archives to be classified, segment the extracted archive titles into words, and use the TFIDF model to calculate the weight value of each word segmentation.
[0009] The cosine similarity function is used to calculate the weighted similarity between the title of the file to be classified and the titles in the file category template library. Based on the similarity, the most similar title in the template library is determined, and the file category to which the most similar title in the template library belongs is output as the title category of the file to be classified.
[0010] As a further step in the automatic matching and classification method for digital personnel file titles in this invention, the existing collected personnel files are classified, and a file category template library is established based on the mapping relationship between the title of each personnel file and its file category.
[0011] As a method for automatic matching and classification of digital personnel file titles in this invention, the corpus is further constructed by word segmentation and encoding. First, the file titles in the file category template library are segmented using the jieba word segmentation library and the constructed prefix dictionary. The segmented word items are used to construct a vector space, and all word items are listed and encoded. Then, the corpus is constructed based on the word items and their encoding.
[0012] As a method for automatic matching and classification of digital personnel file titles in this invention, each element in the corpus is further represented as (word segmentation code, word frequency).
[0013] As a further step in the automatic matching and classification method for digital personnel file titles of this invention, the formula for calculating the weight value of each word segmentation using the TFIDF model is expressed as follows: Where m is the number of times the feature word t appears in the file title text, s is the total number of file titles in the template library, n is the number of file title texts containing the feature word t, and N is the number of file titles.
[0014] As a further step in the present invention, the method for automatically matching and classifying the titles of digital personnel files is as follows: First, OCR text recognition technology is used to extract the title text of the files to be classified. Then, the extracted title text is preprocessed to unify the text character type to the character type in the file category template library. Next, the preprocessed title text is processed by word segmentation and weight value calculation.
[0015] As part of the automatic matching and classification method for digital personnel file titles in this invention, the formula for calculating the word segmentation weight value of the file title text to be classified is expressed as follows: Wherein, IDF(d,t) represents the IDF value of feature word t in archive category d; N represents the total number of archive titles; A represents the number of archive titles in archive category d that contain feature word t; B represents the number of archive titles that contain feature word t but do not belong to archive category d; C represents the number of archive titles in archive category d that do not contain feature word t; and D represents the number of archive titles that do not belong to archive category d and do not contain feature word t.
[0016] As a further step in the automatic matching and classification method for digital personnel file titles of this invention, the formula for calculating the weighted similarity using the cosine similarity function is expressed as follows: Where i represents the sequence of segmented terms after word segmentation of the archive title text, n represents the number of identical segmented terms after word segmentation of the archive title to be classified and the template archive title, and x represents the sequence of segmented terms. i The TFIDF weights of words after word segmentation of the title of the archive to be classified, y i The TFIDF weights of words after word segmentation for the title of the template library.
[0017] Furthermore, the present invention also provides an automatic matching and classification system for digital personnel file titles, comprising: a template library processing module, a target document processing module, and a similarity matching module, wherein,
[0018] The template library processing module is used to establish a file category template library based on the mapping relationship between the title of each personnel file and its file category; and to vectorize the personnel file titles in the file category template library, to create a corpus through word segmentation encoding, and to use the TFIDF model to store the word segmentation weight value of each title.
[0019] The target document processing module is used to extract the title information of the archives to be classified, process the extracted archive titles into words, and use the TFIDF model to calculate the weight value of each word segmentation.
[0020] The similarity matching module is used to calculate the weighted similarity between the title of the file to be classified and the titles in the file category template library using the cosine similarity function. Based on the similarity, the module determines the most similar title in the template library and outputs the file category to which the most similar title belongs as the title category of the file to be classified.
[0021] The beneficial effects of this invention are:
[0022] This invention employs the TFIDF algorithm and cosine similarity algorithm to achieve text matching and automatic classification and archiving of digital personnel file titles. By setting up a template library of mapping relationships between file titles and their respective categories, the title to be classified is matched with the titles in the template library for similarity. The category of the title in the template library with the highest similarity is selected as the category of the title to be classified, thereby achieving automatic classification and archiving of digital personnel file titles. This avoids the problems of low efficiency and human error in the manual classification and archiving of personnel files, which not only greatly improves the efficiency of digital personnel file classification and archiving, but also effectively saves manpower, material resources, and financial resources, making it convenient for application in practical scenarios. Attached Figure Description
[0023] Figure 1 This is a schematic diagram of the automatic matching and classification process for digital personnel file titles in the example.
[0024] Figure 2 This is a schematic diagram illustrating the principle of the automatic matching and classification algorithm in the example.
[0025] Figure 3 This is a template library illustration in the example;
[0026] Figure 4 This is a schematic diagram of TFIDF modeling in the embodiment;
[0027] Figure 5 This is a schematic diagram of the TFIDF process in the embodiment;
[0028] Figure 6 This is a schematic diagram of the improved IDF process in the embodiment;
[0029] Figure 7 This is a schematic diagram of the similarity matching process in the embodiment;
[0030] Figure 8 This is an example illustrating the automatic classification of digital archive titles in the embodiments. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.
[0032] To address the problems of low efficiency, high cost, and susceptibility to errors in manual classification of digitized archives, the embodiments of this invention are described below. Figure 1 As shown, a method for automatically matching and classifying titles in digital personnel files is provided, including:
[0033] S101. Based on the mapping relationship between the title of each personnel file and its corresponding file category, establish a file category template library;
[0034] S102. Vectorize the personnel file titles in the file category template library, and create a corpus by word segmentation encoding. Use the TFIDF model to store the word segmentation weight value of each title.
[0035] S103. Extract the title information of the archives to be classified, process the extracted archive titles into words, and use the TFIDF model to calculate the weight value of each word segmentation.
[0036] S104. Calculate the weighted similarity between the title of the archive to be classified and the titles in the archive category template library using the cosine similarity function. Determine the most similar title in the template library based on the similarity and output the archive category to which the most similar title in the template library belongs as the title category of the archive to be classified.
[0037] First, a TFIDF model is constructed using template library titles. Then, the TFIDF algorithm is used to calculate the weight values of the titles after word segmentation. The titles to be classified and the template library titles are then matched for similarity. The category corresponding to the template library title with the highest similarity is selected as the category of the title to be classified. This solution is simple and easy to implement, avoiding the inefficiencies and human biases inherent in manual classification and archiving of personnel files. It not only significantly improves the efficiency of digital personnel file classification and archiving but also effectively saves manpower, material resources, and financial resources, facilitating its application in practical digital file management scenarios.
[0038] As a preferred embodiment, further, in the process of creating a corpus through word segmentation and encoding, firstly, the file titles in the file category template library are segmented using the jieba word segmentation library and the constructed prefix dictionary. The segmented word items after the file titles are segmented form a vector space, all word items are listed and encoded, and then the corpus is created based on the word items and their encoding.
[0039] See Figure 2 As shown, to address the problem of inconsistent classification standards caused by errors in manual file classification, this embodiment establishes a mapping relationship between the title of each file and its category, forming a category template library. File titles are matched and automatically classified based on this template library, ensuring consistent classification standards and reducing biases caused by subjective judgment. Furthermore, to address the difficulty of directly inputting strings into machine learning models for computation, this embodiment first vectorizes the personnel file titles. The title text in the template library is then segmented using the jieba word segmentation library, listing and encoding all words to create a corpus. A TFIDF model is then built on the corpus to store the segmentation weight values of each title in the template library. To distinguish the categories between title vectors, this embodiment employs the TFIDF algorithm. If a feature word appears frequently in one title but rarely in others, the less frequent the word is considered to have better category discrimination ability. Furthermore, regarding the title information of the archives to be classified, firstly, OCR text recognition technology is used to extract the title text. Then, the extracted title text is preprocessed to unify the character types of the text to the character types in the archive category template library. Next, the preprocessed title text is segmented and weighted. OCR extracts the title information of the archives to be classified, and after segmenting the title into words, the TFIDF model is used to calculate the weight values. Keywords that appear in this title but rarely appear in other titles are assigned higher weight values, thus increasing the weight difference between title text vectors.
[0040] To address the drawback of the TFIDF algorithm in underestimating the weights of frequently occurring feature words, this embodiment utilizes an improved IDF formula to optimize the weight values in the model and the weight values of the titles of the files to be classified. When the number of titles containing feature word 't' is large in one type of file and small in another, it indicates that the frequently occurring feature word 't' has good discriminative power. The improved IDF formula will assign a higher weight to feature word 't', enhancing its discriminative power. Since all personnel file titles fall within the category of short texts, this embodiment uses a cosine similarity algorithm to calculate the similarity between the title to be classified and the titles in the template library, thereby performing title matching. The most similar template library title is matched to the title to be classified, and the category corresponding to this template library title is the category to which the title to be classified belongs.
[0041] Furthermore, based on the above method, this embodiment of the invention also provides an automatic matching and classification system for digital personnel file titles, comprising: a template library processing module, a target document processing module, and a similarity matching module, wherein,
[0042] The template library processing module is used to establish a file category template library based on the mapping relationship between the title of each personnel file and its file category; and to vectorize the personnel file titles in the file category template library, to create a corpus through word segmentation encoding, and to use the TFIDF model to store the word segmentation weight value of each title.
[0043] The target document processing module is used to extract the title information of the archives to be classified, process the extracted archive titles into words, and use the TFIDF model to calculate the weight value of each word segmentation.
[0044] The similarity matching module is used to calculate the weighted similarity between the title of the file to be classified and the titles in the file category template library using the cosine similarity function. Based on the similarity, the module determines the most similar title in the template library and outputs the file category to which the most similar title belongs as the title category of the file to be classified.
[0045] To verify the effectiveness of this solution, the following explanation is provided using a real-world case:
[0046] See Figure 2 The steps shown involve constructing a category template library for personnel file titles, preprocessing the personnel file titles extracted by OCR, segmenting them into words, vectorizing the word frequencies, calculating the similarity between titles using the TFIDF model and cosine function, matching the title with the highest similarity to the category in the template library, and realizing the automatic classification and archiving of digital personnel file titles.
[0047] In this case, according to the regulations in the relevant archival work regulations, the existing personnel files are classified, the mapping relationship between the title of each file and the file category to which it belongs is established, and this mapping relationship is constructed into a text-type template library. For example Figure 3 As shown, since the number and forms of the titles corresponding to the ten major categories of files are numerous, some file categories and some of their corresponding titles are listed here for illustration. Given the template title set s = {s 11 , s 12 , … s ij}, s ij represents the j-th template title in the i-th category of files.
[0048] The titles in the template library are segmented to form a vector space, the segmented words are encoded, a corpus is made and a TFIDF model is constructed. In the TFIDF algorithm, TF represents the term frequency, that is, the frequency of a given word appearing in the title. Among them, IDF is the inverse document frequency, which represents the reciprocal of the number of times a certain feature word appears in all the titles of the entire template library (this index is used to reduce the importance of common words). The specific calculation formula of the TFIDF algorithm is:
[0049]
[0050] Among them, m is the number of times the feature word t appears in the title text, s is the total number of titles in the template library, n is the number of title texts containing the feature word t, adding 0.01 is to prevent the denominator from being 0, and N is the number of titles. Given the template title set s = {s 11 , s 12 , … s ij}, the vector space formed after segmenting the titles is v,
[0051] v = {x1, x2, … x n}, where x i is the segmented term. The segmented terms are encoded to make a corpus. The corpus is represented as (segmented word, term frequency). After that, a TFIDF model is constructed for the corpus, as Figure 4 shown.
[0052] OCR extracts the title information of the file to be classified. Since only the Arabic numeral version of the file title with numbers is defined in the template library, for example, "1994 Annual Assessment Registration Form". Therefore, first preprocess the extracted title to be classified. If the title contains Chinese numerals, preprocess them into Arabic numerals before segmenting. For example, process "one thousand nine hundred and ninety-four" into "1994", so that any form of numbers can be correctly matched with the numbers in the corpus. Input the title s1' to be classified: "one thousand nine hundred and ninety-four annual assessment registration form". After jieba segmentation, the vector space v1' is formed, as Figure 5As shown, the TFIDF weight values of the title s1′ after word segmentation are calculated using the TFIDF model.
[0053] The weight values are optimized using an improved IDF formula. The main idea of IDF is that the fewer the number of headings containing feature term 't' (i.e., the smaller 'n'), the larger the IDF, indicating better discriminative power of feature term 't'. However, if a feature term appears frequently in a particular category of documents—for example, the term "graduate student" appears frequently in the fourth category of academic materials—then 'n' will also be large, and the IDF value obtained according to the IDF formula will be small, indicating weak discriminative power of this feature term. In reality, a feature term appearing frequently in a particular category indicates that it effectively represents that type of document, and such a feature term should be assigned a higher weight. Conversely, if a feature term is evenly distributed across each category of documents, its contribution to classification is smaller. Even if the number of headings containing feature term 't' (n) is small, it should be assigned a smaller weight, but the IDF assigns a large value, which is clearly unreasonable. Therefore, the improved IDF formula can be expressed as follows:
[0054]
[0055] Where IDF(d,t) represents the IDF value of feature word t in document category d; N represents the total number of titles; A represents the number of titles in document category d that contain feature word t; B represents the number of titles that contain feature word t but do not belong to document category d; C represents the number of titles in document category d that do not contain feature word t; and D represents the number of titles that do not belong to document category d and do not contain feature word t.
[0056] Assume the total number of titles is N, and the number of titles for document category d is N. d Then the number of files that do not belong to file category d is N. d We can conclude that: A + C = N d B+D=NN d The modified formula of the improved IDF algorithm can then be expressed as:
[0057]
[0058] Where N, N dGiven values A and B, where A and B represent the number of titles containing the keyword t in the two categories of archives respectively, and are independent of each other, it can be seen that the IDF increases with increasing A, and decreases with increasing B. The improved IDF formula effectively reflects this idea: the more a keyword appears in one category of archives and the less it appears in another, the higher its weight should be assigned. For example, "graduate student" appears frequently in the fourth category of academic credentials but rarely in other categories, so the optimized term "graduate student" should be assigned a higher weight. This applies to the template title set s and the title s2 to be classified. ′ The optimized weighting of the "Registration Form for Master's Degree Candidates" is as follows: Figure 6 As shown, the weight value of the feature word "graduate student" encoded as 2 within the red box is significantly increased, proving that this word can well represent this type of file and has a strong distinguishing ability.
[0059] The process of similarity matching between the title s2′ to be classified and the title set s in the template library is as follows: Figure 7 As shown, the cosine similarity function is used to calculate the similarity between the title to be classified and the titles in the template library. The cosine similarity function can be expressed as:
[0060]
[0061] Where i is the word sequence after word segmentation of the title text, n is the number of identical words between the title to be classified and the title in the template library after word segmentation, and x i The TFIDF weights of words after word segmentation of the title to be classified, y i The TFIDF weights of words after word segmentation for the title of the template library.
[0062] Because the TFIDF algorithm assigns higher weight to keywords that appear less frequently, even if there are very similar titles in the template library, such as "Registration Form for Master's Degree Candidates" and "Registration Form for Doctoral Degree Candidates," which differ by only one word, similar titles can be used to distinguish between keywords that appear less frequently. Figure 8 As shown, the solution in this case can also match the title "Registration Form for Pursuing a Master's Degree" in the template library with the highest similarity, and map it to its category in the template library, thus completing the automatic classification of digital archive titles.
[0063] Unless otherwise specifically stated, the relative steps, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of the invention.
[0064] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0065] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.
[0066] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.
[0067] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for automatically matching and classifying titles in digital personnel files, characterized in that, Includes the following content: Based on the mapping relationship between the title of each personnel file and its corresponding file category, a file category template library is established; The personnel file titles in the file category template library are vectorized, and a corpus is created by word segmentation encoding. The TFIDF model is used to store the word segmentation weight value of each title. Extract the title information of the archives to be classified, segment the extracted titles into words, and calculate the weight value of each word using the TFIDF model. The formula for calculating the weight value of the title text of the archives to be classified is as follows: IDF(d,t) represents the IDF value of feature word t in document category d; N represents the total number of document titles; A represents the number of document titles in document category d that contain feature word t; B represents the number of document titles that contain feature word t but do not belong to document category d; C represents the number of document titles in document category d that do not contain feature word t; D represents the number of document titles that do not belong to document category d and do not contain feature word t. The cosine similarity function is used to calculate the weighted similarity between the title of the file to be classified and the titles in the file category template library. Based on the similarity, the most similar title in the template library is determined, and the file category to which the most similar title in the template library belongs is output as the title category of the file to be classified.
2. The method for automatic matching and classification of digital personnel file titles according to claim 1, characterized in that, The existing personnel files are classified, and a file category template library is established based on the mapping relationship between the title of each personnel file and its file category.
3. The method for automatic matching and classification of digital personnel file titles according to claim 1, characterized in that, In constructing a corpus through word segmentation and encoding, firstly, the file titles in the file category template library are segmented using the jieba word segmentation library and a constructed prefix dictionary. The segmented word items are used to construct a vector space, and all word items are listed and encoded. Then, the corpus is constructed based on the word items and their encodings.
4. The method for automatically matching and classifying titles of digital personnel files according to claim 1 or 3, characterized in that, Each element in the corpus is represented as (word segmentation code, word frequency).
5. The method for automatic matching and classification of digital personnel file titles according to claim 1, characterized in that, The formula for calculating the weight value of each word segmentation using the TFIDF model is expressed as follows: Where m is the number of times the feature word t appears in the file title text, s is the total number of file titles in the template library, n is the number of file title texts containing the feature word t, and N is the number of file titles.
6. The method for automatic matching and classification of digital personnel file titles according to claim 1, characterized in that, For the title information of the archives to be classified, firstly, OCR text recognition technology is used to extract the title text of the archives to be classified. Then, the extracted title text is preprocessed to unify the text character type to the character type in the archive category template library. Next, the preprocessed title text is processed by word segmentation and weight value calculation.
7. The method for automatic matching and classification of digital personnel file titles according to claim 6, characterized in that, The formula for calculating weighted similarity using the cosine similarity function is expressed as follows: Where i represents the sequence of segmented terms after word segmentation of the archive title text, n represents the number of identical segmented terms after word segmentation of the archive title to be classified and the template archive title, and x represents the sequence of segmented terms. i The TFIDF weights of words after word segmentation of the title of the archive to be classified, y i The TFIDF weights of words after word segmentation for the title of the template library.
8. A digital personnel file title automatic matching and classification system, characterized in that, It includes: a template library processing module, a target document processing module, and a similarity matching module. The template library processing module is used to establish a file category template library based on the mapping relationship between the title of each personnel file and its file category; and to vectorize the personnel file titles in the file category template library, to create a corpus through word segmentation encoding, and to use the TFIDF model to store the word segmentation weight value of each title. The target document processing module is used to extract the title information of the archives to be classified, segment the extracted archive titles into words, and calculate the weight value of each word using the TFIDF model. The formula for calculating the weight value of the title text of the archives to be classified is expressed as follows: IDF(d,t) represents the IDF value of feature word t in document category d; N represents the total number of document titles; A represents the number of document titles in document category d that contain feature word t; B represents the number of document titles that contain feature word t but do not belong to document category d; C represents the number of document titles in document category d that do not contain feature word t; D represents the number of document titles that do not belong to document category d and do not contain feature word t. The similarity matching module is used to calculate the weighted similarity between the title of the file to be classified and the titles in the file category template library using the cosine similarity function. Based on the similarity, the module determines the most similar title in the template library and outputs the file category to which the most similar title belongs as the title category of the file to be classified.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.