Electronic archive management method and system based on big data
Through the electronic archive management method based on big data, including word segmentation extraction, importance calculation and knowledge graph expansion, the problem of low classification accuracy of electronic archives is solved, and higher classification accuracy and error recognition capabilities are achieved.
Patent Information
- Application Number
- CN202411908896.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-04-15
AI Technical Summary
The prior art is difficult to ensure the accuracy of staff members when classifying electronic files, which makes it difficult to avoid classification errors.
Using the electronic archive management method based on big data, the initial word set is obtained through word segmentation extraction and vectorization, the first similarity of word segmentation is calculated, and a directed graph is constructed to calculate the importance, and the important word set is selected. Then, the entity word segmentation set and background word segmentation set are obtained through the knowledge graph, the second similarity is calculated to expand the word set, and finally the electronic archives that may be classified errors are identified by calculating the classification accuracy.
It improves the accuracy of electronic file classification, identifys electronic files that may be classified incorrectly, and reduces the burden of secondary identification calibration for staff.
Smart Images
Figure CN119830905A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electronic archive management, and in particular to an electronic archive management method and system based on big data. Background Art
[0002] Shandong Province is home to a large number of ethnic minorities. The ethnic customs and personal experiences of different individuals have distinct ethnic characteristics. For example, there are dozens of categories based on ethnicity alone. Therefore, personal electronic files are extremely diverse. When managing electronic files, they need to be classified and stored. Moreover, due to the diversity of electronic file data, in order to facilitate later retrieval, there are relatively many classification index titles. Under so many classification index titles, staff need to accurately classify electronic files into different classification index titles. This is also a major challenge for staff and is extremely difficult. It is difficult to avoid incorrect classification of electronic files. Summary of the invention
[0003] In view of the deficiencies of the prior art, the present invention provides an electronic archive management method and system based on big data, which solves the problem in the prior art that it is difficult to ensure the accuracy of staff when classifying electronic archives.
[0004] To achieve the above object, the present invention provides the following technical solutions:
[0005] A method for managing electronic archives based on big data, the method comprising the following steps:
[0006] S1. Obtain the electronic archive source data to be processed based on big data, and perform word segmentation extraction on the electronic archive source data to obtain an initial word set of each electronic archive source data, and vectorize it;
[0007] S2, calculating the first similarity of each word in the initial word set, and constructing a directed graph of the initial word set to calculate the importance of each word, so as to obtain an important word set containing several words with the greatest importance;
[0008] S3, obtaining the entity segmentation set of each segmentation in the important word set through the knowledge graph, and calculating the second similarity of the context containing the entity segmentation set and the corresponding segmentation in the important word set, and adding the segmentation corresponding to the context of the entity segmentation set with the largest second similarity to the important word set to obtain the extended word set;
[0009] S4. Obtain several set classification words, and calculate the probability of each word in each electronic archive source data under the same classification word in the remaining electronic archive source data to obtain the classification accuracy, so as to identify the electronic archive source data that does not belong to each classification word.
[0010] Preferably, in step S2, the following steps are specifically included:
[0011] S21, setting a statistical window including several segmented words in the initial word set with each segmented word in the initial word set as the center;
[0012] S22, taking each word in the initial word set as a vertex and constructing edges between the words included in the same statistical window to obtain a directed graph of the initial word set;
[0013] S23, calculating the first similarity of each word in the initial word set; the calculation formula of the first similarity is:
[0014]
[0015] In the above formula, represents the first similarity between the i-th word and the j-th word in the initial word set, w i and w j Respectively represent the vectors of the i-th and j-th word segments in the initial word set;
[0016] S24, calculating the importance of each participle in the initial word set according to the directed graph and the first similarity of each participle;
[0017] S25, select the words in the initial word sets with the largest importance values as the important word sets; the expression of the important word sets is:
[0018] ZWC(a)={Zw1, Zw2,…,Zw B}
[0019] In the above formula, ZWC(a) represents the important word set of the a-th electronic archive source data, w B Represents the Bth word in the important word set.
[0020] Preferably, in step S21, the following steps are specifically included:
[0021] S211, setting the lengths of several statistical windows with each word in the initial word set as the center, wherein the length of the statistical window is the number of words in the initial word set;
[0022] S212, setting a test set to test steps S1 to S4, and obtaining the test results;
[0023] S213, calculating the F1 value according to the test results of step S1 to step S4, and constructing an F1 value curve;
[0024] S214, determining whether there is a peak point in the F1 value curve;
[0025] If it exists, go to step S215;
[0026] If not, return to step S211;
[0027] S215, determining whether the difference in length of the statistical window corresponding to the peak point in the F1 value curve and the two adjacent F1 value points is 1;
[0028] If yes, proceed to step S216;
[0029] If not, reset the length of the statistical window and return to step S212;
[0030] S216. Select the length when the F1 value is the largest as the length of the statistical window.
[0031] Preferably, in step S215, the method for resetting the lengths of a plurality of statistical windows specifically comprises the following steps:
[0032] S2151, taking the length of the statistical window corresponding to two F1 value points adjacent to the peak point in the F1 value curve as the initial point;
[0033] S2152, calculating the absolute value of the difference between the two initial points to obtain the window distance;
[0034] S2153, determining whether the length of the statistical window to be set is less than the window distance;
[0035] If yes, proceed to step S5154;
[0036] If not, all integers between the two initial points are set as the length of the statistical window, and then the process returns to step S212;
[0037] S2154. Generate a number of integers with different values between the two initial points to serve as the length of the statistical window.
[0038] Preferably, in step S3, the following steps are specifically included:
[0039] S31. Obtain the entity segmentation set of each segmentation in the important word set through the knowledge graph; the expression of the entity segmentation set is:
[0040] ST(Zw B )={Zw B d1, Zw B d2,…,Zw B d D}
[0041] In the above formula, ST(Zw B ) represents the entity segmentation set of the Bth segmentation of the important word set obtained through the knowledge graph, Zw B dD is the Dth word in the entity word set;
[0042] S32, extracting the background text of each word in the entity word set according to the knowledge graph to obtain the background word set of each word; the expression of the background word set is:
[0043] BJ(Zw B d D ) = {BJ1, BJ2, ..., BJ C}
[0044] In the above formula, BJ(Zw B d D ) represents the background segmentation set of the Dth segmentation in the entity segmentation set, BJ C Represents the cth word in the background word set;
[0045] S33, obtaining a text set of several sequentially connected word segments and including any word segment in the important word set from the initial word set to form an important text word set; the expression of the important text word set is:
[0046] BJ(Zw B )={w c-f ,…,w c-1 , w c , w c+1 , …w c+F , w c+F+1}
[0047] In the above formula, BJ(Zw B ) represents the Bth participle Zw in the important word set B The important text words set, w c is the cth word in the initial word set, the Bth word in the important word set is the same as the cth word in the initial word set, w c+F and w c-f represents the Fth and fth participles adjacent to the cth participle in the initial word set;
[0048] S34, respectively calculating the second similarity between each background segmentation set of the Dth segmentation in the entity segmentation set and the important text word set corresponding to the segmentation in the important word set; the calculation formula of the second similarity is:
[0049]
[0050] In the above formula, S 2 (BJ(Zw B ), BJ(Zw B d D )) represents the important text word set BJ(Zw B) and the background word set BJ(Zw B d D ), and They are the important text word sets BJ(Zw B ) and the background segmentation set BJ(Zw B d D ) vector;
[0051] S35. Add the background word set with the second largest similarity to the important word set to obtain an expanded word set.
[0052] Preferably, in step S4, the following steps are specifically included:
[0053] S41, obtaining the expanded word set corresponding to each electronic archive source data under each classification word;
[0054] S42, obtaining the number of occurrences of each participle in each expanded word set under the same classifier;
[0055] S43, marking the expanded word set corresponding to each electronic archive source data as a word set to be classified, and marking the expanded word set corresponding to the remaining electronic archive source data belonging to the same classification word as a processed word set;
[0056] S44, calculating the classification accuracy of the word set to be classified according to whether each word in the word set to be classified appears in the processed word set and the number of times each word appears in the word set to be classified and the processed word set respectively;
[0057] S45, setting an accuracy threshold, and determining whether the classification accuracy of the word set to be classified is greater than the accuracy threshold;
[0058] If so, then end;
[0059] If not, proceed to step S46;
[0060] S46, assigning the word set to be classified to other classification words in turn, and repeating steps S41-S44 to obtain several classification accuracy rates;
[0061] S47, outputting a number of classification words corresponding to the maximum classification accuracy, and then ending.
[0062] Preferably, in step S44, the following steps are specifically included:
[0063] S441, respectively calculating the repetition rate of each word in the word set to be classified in each processed word set; the calculation formula of the repetition rate is:
[0064]
[0065] In the above formula, represents the repetition rate of the αth word in the word set to be classified, represents the number of times the αth word in the word set to be classified appears in the processed word set, and N' represents the number of processed word sets under the same classification word as the word to be classified;
[0066] S442, calculating the classification accuracy according to the repetition rate of each word in the word set to be classified; the calculation formula of the classification accuracy is:
[0067]
[0068] In the above formula, P represents the classification accuracy of the word set to be classified, represents the repetition rate of the αth word segment in the word set to be classified, and β represents the number of word segments in the word set to be classified.
[0069] Preferably, in step S24, the calculation formula of importance is:
[0070]
[0071] In the above formula, Z i and Z j They represent the importance of the i-th and j-th participles in the initial word set, respectively. A is the correlation coefficient, 0.7≤A≤0.95, and w j represents the jth word in the initial word set, V i 、V j and V k represent the ith, jth, and kth vertices in the directed graph, respectively, and correspond to the ith, jth, and kth word segments in the initial word set, respectively. IN(V i ) represents the set of all vertices pointing to the jth vertex in the directed graph, OUT(V j ) represents the vertex V in the directed graph j The set of all vertices pointed to by .
[0072] Preferably, in step S213, the calculation formula of the F1 value is:
[0073]
[0074] In the above formula, F1 represents the F1 value, R represents the recall rate of the training result, and P represents the precision rate of the training result.
[0075] The technical solution also provides a system for implementing an electronic archive management method based on big data, the system comprising: a processor and a memory, the memory being used to store a computer program, and the computer program implementing the electronic archive management method when executed by the processor.
[0076] Compared with the prior art, the present invention provides an electronic archive management method and system based on big data, which has the following beneficial effects:
[0077] 1. The present invention obtains all electronic archives in the archive management system, then performs word segmentation extraction and vectorization on the electronic archives to obtain an initial word set, and calculates the first similarity of each word in the initial word set and constructs a directed graph to select the important word set that can best reflect the core content of the electronic archive by calculating the importance, and then clarifies the meaning of each word in the important word set according to the knowledge graph to eliminate ambiguity, thereby obtaining an expanded word set, and finally calculates the classification accuracy of each electronic archive source data to determine whether each electronic archive source data is wrong.
[0078] 2. The present invention calculates the first similarity to reflect the similarity of each word segment, and constructs a directed graph, through the connection relationship between each vertex and edge in the directed graph, so as to select the word segment in the initial word set that can reflect the core content of the electronic file by calculating the importance, so as to obtain the important word set.
[0079] 3. The present invention calculates the F1 value to obtain the optimal statistical window length among several set window lengths. The optimal statistical window length corresponds to the maximum F1 value and is located on the peak value, thereby maximizing the calculation accuracy of the method, and the method can automatically adjust and select the optimal statistical window length accordingly.
[0080] 4. The present invention constructs a background word segmentation set based on the knowledge graph, and compares it with the important text set to calculate the second similarity, so as to select the background word segmentation set that best reflects the meaning of each word in the important word set, and add the background word segmentation set to the important word set to obtain an expanded word set, thereby maximally eliminating the ambiguity of each word in the important word set.
[0081] 5. The present invention sequentially marks the expanded word sets corresponding to the electronic archive source data under the same classification word as the word sets to be classified, and marks the expanded word sets corresponding to the remaining electronic archive source data as the classified word sets, thereby calculating the accuracy of each word set to be classified to determine whether each electronic archive source data is misclassified, thereby identifying the electronic archive source data with classification errors in the archive system for the staff to perform secondary recognition and calibration. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0083] Figure 1The figure is a flow chart of the electronic archive management method based on big data of the present invention. DETAILED DESCRIPTION
[0084] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.
[0085] Those skilled in the art can understand that all or part of the steps in the following embodiments can be completed by instructing the relevant hardware through a program, so the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0086] In order to solve the problem in the prior art that it is difficult to ensure the accuracy of staff in classifying electronic files, the present invention provides an electronic file management method based on big data, which manages electronic files in the file management system and identifies electronic files that may be misclassified, so that staff can conduct a secondary check, thereby improving the accuracy of electronic file classification. The management method includes the following steps:
[0087] S1. Based on big data, the electronic archive source data to be processed is obtained, and word segmentation and extraction are performed on the electronic archive source data to obtain the initial word set of each electronic archive source data, and vectorization is performed; the word segmentation and vectorization methods mainly include Word2Vec method, BERT model and VSM model, etc. The expression of the initial word set is:
[0088] WC(a)={w1,w2,…,w B
[0089] In the above formula, WC(a) represents the initial word set of the a-th electronic archive source data, w B represents the Bth word in the initial word set;
[0090] S2, calculate the first similarity of each word in the initial word set, and construct a directed graph of the initial word set to calculate the importance of each word, so as to obtain an important word set containing several words with the greatest importance; the initial word set after word segmentation extraction contains a large number of words, and at this time, it is necessary to calculate the importance of each word according to the degree of association and similarity between each word and other words, so as to select the words that best reflect the core content of the electronic archive according to the importance to obtain an important word set. In step S2, the following steps are specifically included:
[0091] S21, respectively setting a statistical window including several word segments in the initial word set with each word segment in the initial word set as the center; in order to further describe the setting method and specific value of the statistical window, and after the initial setting is completed, the system can automatically improve and adjust the setting of the lengths of several statistical windows according to the calculation situation, and select the length of the final statistical window, in step S21, specifically including the following steps:
[0092] S211, setting the lengths of several statistical windows with each word in the initial word set as the center, wherein the length of the statistical window is the number of words in the initial word set;
[0093] S212, setting a test set to test steps S1 to S4, and obtaining the test results;
[0094] S213, calculating the F1 value according to the test results of step S1 to step S4, and constructing an F1 value curve;
[0095]
[0096] In the above formula, F1 represents the F1 value, R represents the recall rate of the training result, and P represents the precision rate of the training result;
[0097] S214, judging whether there is a peak point in the F1 value curve; the identification of the peak point can be assisted by MATALB software;
[0098] If it exists, go to step S215;
[0099] If not, return to step S211;
[0100] S215, determining whether the difference in length of the statistical window corresponding to the peak point in the F1 value curve and the two adjacent F1 value points is 1;
[0101] If yes, proceed to step S216;
[0102] If not, reset the length of the statistical window and return to step S212;
[0103] In step S215, in order to enable the system to automatically change to the optimized statistical window length when the length of the statistical window initially set manually is inappropriate, save manpower, and reset the lengths of several statistical windows, the method of resetting the lengths of several statistical windows specifically includes the following steps:
[0104] S2151, taking the length of the statistical window corresponding to the two F1 value points adjacent to the peak point in the F1 value curve as the initial point; for example, if the lengths of the two statistical windows are 2 and 10 respectively, these two values are set as the initial points, and the window distance calculated in step S2152 is 8. At this time, if the number of window lengths set is 5, 5 integer values between 2 and 8 are randomly generated as the lengths of several newly set statistical windows, so as to finally obtain the lengths of several statistical windows whose numerical values are connected in sequence, so as to more accurately obtain the most suitable length of the statistical window;
[0105] S2152, calculating the absolute value of the difference between the two initial points to obtain the window distance;
[0106] S2153, determining whether the length of the statistical window to be set is less than the window distance;
[0107] If yes, proceed to step S5154;
[0108] If not, all integers between the two initial points are set as the length of the statistical window;
[0109] S2154. Generate a number of integers of different values between the two initial points as the length of the statistical window, and then return to step S212.
[0110] S216. Select the length when the F1 value is the largest as the length of the statistical window.
[0111] S22, taking each word in the initial word set as a vertex and constructing edges between the words included in the same statistical window to obtain a directed graph of the initial word set;
[0112] S23, calculating the first similarity of each word in the initial word set; the calculation formula of the first similarity is:
[0113]
[0114] In the above formula, represents the first similarity between the i-th word and the j-th word in the initial word set, w i and w j Respectively represent the vectors of the i-th and j-th word segments in the initial word set;
[0115] S24, calculating the importance of each participle in the initial word set according to the directed graph and the first similarity of each participle;
[0116]
[0117] In the above formula, Z i and Z jThey represent the importance of the i-th and j-th participles in the initial word set, respectively. A is the correlation coefficient, 0.7≤A≤0.95, and w j represents the jth word in the initial word set, V i 、V j and V k represent the ith, jth, and kth vertices in the directed graph, respectively, and correspond to the ith, jth, and kth word segments in the initial word set, respectively. IN(V i ) represents the set of all vertices pointing to the jth vertex in the directed graph, OUT(V j ) represents the vertex V in the directed graph j The set of all vertices pointed to;
[0118] S25, select the words in the initial word sets with the largest importance values as the important word sets; the expression of the important word sets is:
[0119] ZWC(a)={Zw1, Zw2,…,Zw B}
[0120] In the above formula, ZWC(a) represents the important word set of the a-th electronic archive source data, w B Represents the Bth word in the important word set.
[0121] S3, obtaining the entity segmentation set of each segmentation in the important word set through the knowledge graph, and calculating the second similarity of the context containing the entity segmentation set and the corresponding segmentation in the important word set, and adding the segmentation corresponding to the context of the entity segmentation set with the largest second similarity to the important word set to obtain the extended word set; in Chinese, a word may have different meanings, such as the word "tree", which may mean trees growing in real life, or it may mean a tree structure in the computer field, so it is necessary to accurately identify the specific meaning of each segmentation to avoid ambiguity. In step S3, the following steps are specifically included:
[0122] S31. Obtain the entity segmentation set of each segmentation in the important word set through the knowledge graph; the knowledge graph used is CN-DBpedia, etc., and the expression of the entity segmentation set is:
[0123] ST(Zw B )={Zw B d1, Zw B d2,…,Zw B d D}
[0124] In the above formula, ST(Zw B) represents the entity segmentation set of the Bth segmentation of the important word set obtained through the knowledge graph, that is, each segmentation in the important word set may have multiple B different meanings, and the B different meanings are obtained through the knowledge graph. Zw B d D is the Dth word in the entity word set;
[0125] S32, extract the background text of each segmentation in the entity segmentation set according to the knowledge graph to obtain the background segmentation set of each segmentation; in the knowledge graph, the meaning or application of each segmentation in the entity segmentation set is explained through the background text, so it is necessary to extract the core content in the background text to generate the background segmentation set, and the expression of the background segmentation set is:
[0126] BJ(Zw B d D ) = {BJ1, BJ2, ..., BJ C}
[0127] In the above formula, BJ(Zw B d D ) represents the background segmentation set of the Dth segmentation in the entity segmentation set, BJ C Represents the cth word in the background word set;
[0128] S33, obtaining a text set of several sequentially connected word segments and including any word segment in the important word set from the initial word set to form an important text word set, wherein the important text word set is a practical example of the application of any word segment in the important word set; the expression of the important text word set is:
[0129] BJ(Zw B )={w c-f ,…,w c-1 , w c , w c+1 , …w c+F , w c+F+1}
[0130] In the above formula, BJ(Zw B ) represents the Bth participle Zw in the important word set B The important text words set, w c is the cth word in the initial word set, the Bth word in the important word set is the same as the cth word in the initial word set, w c+F and w c-f represents the Fth and fth participles adjacent to the cth participle in the initial word set;
[0131] S34, respectively calculating the second similarity between each background segmentation set of the Dth segmentation in the entity segmentation set and the important text word set corresponding to the segmentation in the important word set; the calculation formula of the second similarity is:
[0132]
[0133] In the above formula, S 2 (BJ(Zw B ), BJ(Zw B d D )) represents the important text word set BJ(Zw B ) and the background word set BJ(Zw B d D ), and They are the important text word sets BJ(Zw B ) and the background word set BJ(Zw B d D ) vector;
[0134] S35. Add the background word set with the second largest similarity to the important word set to obtain an expanded word set.
[0135] S4, obtaining several set classification words, and calculating the probability of each word in each electronic archive source data under the same classification word in the rest of the electronic archive source data to obtain the classification accuracy, so as to identify the electronic archive source data that does not belong to each classification word. The classification word refers to the title index for classifying several electronic archive source data, such as dividing them into several ethnic groups such as the Han nationality by ethnicity, and each ethnic group is a classification word. In step S4, the following steps are specifically included:
[0136] S41, obtaining the expanded word set corresponding to each electronic archive source data under each classification word;
[0137] S42, obtaining the number of occurrences of each participle in each expanded word set under the same classifier, that is, in which expanded word sets each participle appears respectively;
[0138] S43, marking the expanded word set corresponding to each electronic archive source data as a word set to be classified, and marking the expanded word set corresponding to the remaining electronic archive source data belonging to the same classification word as a processed word set;
[0139] S44, according to whether each word in the word set to be classified appears in the processed word set, and the number of times each word appears in the word set to be classified and the processed word set, the classification accuracy of the word set to be classified is calculated; in order to calculate the degree of association between the word set to be classified and each processed word under the same classification word, the number of occurrences and probability of each word in the word set to be classified in the processed word set are reflected, in step S44, the following steps are specifically included:
[0140] S441, respectively calculating the repetition rate of each word in the word set to be classified in each processed word set; the calculation formula of the repetition rate is:
[0141]
[0142] In the above formula, represents the repetition rate of the αth word in the word set to be classified, represents the number of times the αth word in the word set to be classified appears in the processed word set, and N' represents the number of processed word sets under the same classification word as the word to be classified;
[0143] S442, calculating the classification accuracy according to the repetition rate of each word in the word set to be classified; the calculation formula of the classification accuracy is:
[0144]
[0145] In the above formula, P represents the classification accuracy of the word set to be classified, represents the repetition rate of the αth word segment in the word set to be classified, and β represents the number of word segments in the word set to be classified.
[0146] S45, setting an accuracy threshold, and determining whether the classification accuracy of the word set to be classified is greater than the accuracy threshold;
[0147] If so, then end;
[0148] If not, proceed to step S46;
[0149] S46, assigning the word set to be classified to other classification words in turn, and repeating steps S41-S44 to obtain several classification accuracy rates;
[0150] S47, outputting a number of classification words corresponding to the maximum classification accuracy, and then ending.
[0151] When it is necessary to check whether there are errors in the classification of each electronic archive in the archive management system, by running this method, all electronic archives in the archive management system are first obtained, and then the electronic archives are segmented, extracted and vectorized to obtain an initial word set, and by calculating the first similarity of each word in the initial word set and constructing a directed graph, the important word set that can best reflect the core content of the electronic archive is selected by calculating the importance, and then the meaning of each word in the important word set is clarified according to the knowledge graph to eliminate ambiguity, so as to obtain an extended word set, and finally by calculating the classification accuracy of each electronic archive source data, it is determined whether each electronic archive source data is wrong.
[0152] Corresponding to the electronic archive management method based on big data provided in the above embodiment, this embodiment also provides a system for implementing the electronic archive management method based on big data. Since the electronic archive management system provided in this embodiment corresponds to the electronic archive management method provided in the above embodiment, the implementation method of the aforementioned electronic archive management method is also applicable to the electronic archive management system provided in this embodiment and will not be described in detail in this embodiment.
[0153] The system includes a processor and a memory, wherein the memory is used to store a computer program, and the computer program implements an electronic archive management method when executed by the processor.
[0154] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. An electronic archive management method based on big data, characterized in that: The management method includes the following steps: S1. Obtain the electronic archive source data to be processed based on big data, and perform word segmentation extraction on the electronic archive source data to obtain an initial word set of each electronic archive source data, and vectorize it; S2, calculating the first similarity of each word in the initial word set, and constructing a directed graph of the initial word set to calculate the importance of each word, so as to obtain an important word set containing several words with the greatest importance; S3, obtaining the entity segmentation set of each segmentation in the important word set through the knowledge graph, and calculating the second similarity of the context containing the entity segmentation set and the corresponding segmentation in the important word set, and adding the segmentation corresponding to the context of the entity segmentation set with the largest second similarity to the important word set to obtain the extended word set; S4. Obtain several set classification words, and calculate the probability of each word in each electronic archive source data under the same classification word in the remaining electronic archive source data to obtain the classification accuracy, so as to identify the electronic archive source data that does not belong to each classification word.
2. The electronic archive management method according to claim 1, characterized in that: In step S2, the following steps are specifically included: S21, setting a statistical window including several segmented words in the initial word set with each segmented word in the initial word set as the center; S22, taking each word in the initial word set as a vertex and constructing edges between the words included in the same statistical window to obtain a directed graph of the initial word set; S23, calculating the first similarity of each word in the initial word set; the calculation formula of the first similarity is: In the above formula, represents the first similarity between the i-th word and the j-th word in the initial word set, w i and w j Respectively represent the vectors of the i-th and j-th word segments in the initial word set; S24, calculating the importance of each participle in the initial word set according to the directed graph and the first similarity of each participle; S25, select the words in the initial word sets with the largest importance values as the important word sets; the expression of the important word sets is: ZWC(a)={Zw1,Zw2,…,Zw B In the above formula, ZWC(a) represents the important word set of the a-th electronic archive source data, w B Represents the Bth word in the important word set.
3. The electronic archive management method according to claim 2, characterized in that: In step S21, the following steps are specifically included: S211, setting the lengths of several statistical windows with each word in the initial word set as the center, wherein the length of the statistical window is the number of words in the initial word set; S212, setting a test set to test steps S1 to S4, and obtaining the test results; S213, calculating the F1 value according to the test results of step S1 to step S4, and constructing an F1 value curve; S214, determining whether there is a peak point in the F1 value curve; If it exists, go to step S215; If not, return to step S211; S215, determining whether the difference in length of the statistical window corresponding to the peak point in the F1 value curve and the two adjacent F1 value points is 1; If yes, proceed to step S216; If not, reset the length of the statistical window and return to step S212; S216. Select the length when the F1 value is the largest as the length of the statistical window.
4. The electronic archive management method according to claim 2, characterized in that: In step S215, the method for resetting the lengths of a plurality of statistical windows specifically comprises the following steps: S2151, taking the length of the statistical window corresponding to two F1 value points adjacent to the peak point in the F1 value curve as the initial point; S2152, calculating the absolute value of the difference between the two initial points to obtain the window distance; S2153, determining whether the length of the statistical window to be set is less than the window distance; If yes, proceed to step S5154; If not, all integers between the two initial points are set as the length of the statistical window, and then the process returns to step S212; S2154. Generate a number of integers with different values between the two initial points to serve as the length of the statistical window.
5. The electronic archive management method according to claim 1, characterized in that: In step S3, the following steps are specifically included: S31. Obtain the entity segmentation set of each segmentation in the important word set through the knowledge graph; the expression of the entity segmentation set is: ST(Zw B )={Zw B d1,Zw B d2,…,Zw B d D } In the above formula, ST(Zw B ) represents the entity segmentation set of the Bth segmentation of the important word set obtained through the knowledge graph, Zw B d D is the Dth word in the entity word set; S32, extracting the background text of each word in the entity word set according to the knowledge graph to obtain the background word set of each word; the expression of the background word set is: BJ(Zw B d D )={BJ1,BJ2,…,BJ C } In the above formula, BJ(Zw B d D ) represents the background segmentation set of the Dth segmentation in the entity segmentation set, BJ C Represents the cth word in the background word set; S33, obtaining a text set of several sequentially connected word segments and including any word segment in the important word set from the initial word set to form an important text word set; the expression of the important text word set is: BJ(ZW B )={w c-f ,…,w c-1 ,w c ,w c+1 ,…w c+F ,w c+F+i } In the above formula, BJ(Zw B ) represents the Bth participle Zw in the important word set B The important text words set, w c is the cth word in the initial word set, the Bth word in the important word set is the same as the cth word in the initial word set, W c+F and w c-f represents the Fth and fth participles adjacent to the cth participle in the initial word set; S34, respectively calculating the second similarity between each background segmentation set of the Dth segmentation in the entity segmentation set and the important text word set corresponding to the segmentation in the important word set; the calculation formula of the second similarity is: In the above formula, S 2 (BJ(Zw B ), BJ(Zw B d D )) represents the important text word set BJ(Zw B ) and the background segmentation set BJ(Zw B d D ), and They are the important text word sets BJ(Zw B ) and the background word set BJ(Zw B d D ) vector; S35. Add the background word set with the second largest similarity to the important word set to obtain an expanded word set.
6. The electronic archive management method according to claim 1, characterized in that: In step S4, the following steps are specifically included: S41, obtaining the expanded word set corresponding to each electronic archive source data under each classification word; S42, obtaining the number of occurrences of each participle in each expanded word set under the same classifier; S43, marking the expanded word set corresponding to each electronic archive source data as a word set to be classified, and marking the expanded word set corresponding to the remaining electronic archive source data belonging to the same classification word as a processed word set; S44, calculating the classification accuracy of the word set to be classified according to whether each word in the word set to be classified appears in the processed word set and the number of times each word appears in the word set to be classified and the processed word set respectively; S45, setting an accuracy threshold, and determining whether the classification accuracy of the word set to be classified is greater than the accuracy threshold; If so, then end; If not, proceed to step S46; S46, assigning the word set to be classified to other classification words in turn, and repeating steps S41-S44 to obtain several classification accuracy rates; S47, outputting a number of classification words corresponding to the maximum classification accuracy, and then ending.
7. The electronic archive management method according to claim 6, characterized in that: In step S44, the following steps are specifically included: S441, respectively calculating the repetition rate of each word in the word set to be classified in each processed word set; the calculation formula of the repetition rate is: In the above formula, represents the repetition rate of the αth word in the word set to be classified, represents the number of times the αth word in the word set to be classified appears in the processed word set, and N' represents the number of processed word sets under the same classification word as the word to be classified; S442, calculating the classification accuracy according to the repetition rate of each word in the word set to be classified; the calculation formula of the classification accuracy is: In the above formula, P represents the classification accuracy of the word set to be classified, represents the repetition rate of the αth word segment in the word set to be classified, and β represents the number of word segments in the word set to be classified.
8. The electronic archive management method according to claim 2, characterized in that: In step S24, the calculation formula of importance is: In the above formula, Z i and Z j They represent the importance of the i-th and j-th participles in the initial word set, respectively. A is the correlation coefficient, 0.7≤A≤0.95, and w j represents the jth word in the initial word set, V i 、V j and V k represent the ith, jth, and kth vertices in the directed graph, respectively, and correspond to the ith, jth, and kth word segments in the initial word set, respectively. IN(V i ) represents the set of all vertices pointing to the jth vertex in the directed graph, OUT(V j ) represents the vertex V in the directed graph j The set of all vertices pointed to by .
9. The electronic archive management method according to claim 3, characterized in that: In step S213, the calculation formula of the F1 value is: In the above formula, F1 represents the F1 value, R represents the recall rate of the training result, and P represents the precision rate of the training result.
10. A management system for implementing the electronic archive management method based on big data according to any one of claims 1 to 9, characterized in that: It includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, the electronic archive management method according to any one of claims 1 to 9 is implemented.