Archive data classification method and device based on artificial intelligence and storage medium

Abstract

The invention relates to the technical field of archive classification, in particular to an archive data classification method and device based on artificial intelligence and a storage medium, and the method comprises the steps: collecting a target scanning image and target archive text data, constructing a material fingerprint knowledge graph according to an age paper characteristic graph, and carrying out the matching of a target material fingerprint, performing word segmentation on the target archive text data to obtain a word segmentation set, performing type matching on the word segmentation set and a preset literature word bank, extracting target text features, judging a target text quality state based on a target scanning image, collecting historical archive text data, and updating the target text features. And classifying the target file based on the target material fingerprint and the target text feature, and storing the target file according to the classification. According to the invention, the classification efficiency of the archive data is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of archive classification, and in particular to an archive data classification method, device and storage medium based on artificial intelligence. Background Art

[0002] Traditional archive classification methods mainly rely on manual experience to identify paper materials and text content, which has problems such as low efficiency, strong subjectivity, and susceptibility to individual differences. In the existing technology, although some automated solutions attempt to combine image processing and text analysis, they are mostly limited to single-dimensional features and fail to effectively integrate multispectral image data and deep semantic information, resulting in insufficient classification accuracy. In addition, there is a lack of systematic solutions to problems such as variant character interference and misidentification of low-quality scanned images in text processing, which affects the robustness of the model.

[0003] Therefore, developing an efficient and accurate automatic classification method for archival data has important practical significance. Summary of the invention

[0004] The object of the present invention is to provide an artificial intelligence-based archival data classification method, device and storage medium to solve at least one of the problems existing in the prior art.

[0005] To achieve the above object, according to one aspect of the present application, the present invention provides an artificial intelligence-based archival data classification method, comprising: Collect target scanned images and target archive text data; Construct a material fingerprint knowledge graph based on the paper feature graph of the era, and match the target material fingerprint based on the material fingerprint knowledge graph and the collected target scan image; Segment the target archive text data to obtain a segmented word set, then match the segmented word set with a preset document word library, and extract target text features based on the type matching results; The target text quality status is judged based on the target scanned image, and historical archive text data is collected, and then the target text features are updated based on the target text quality status judgment result and the historical archive text data.

[0006] Optionally, feature collection is performed on paper feature images of each era according to preset bands to obtain feature vectors of each preset band; Then, the feature vectors of each preset band of the paper feature images of each era are used as the material fingerprint knowledge graph.

[0007] Optionally, based on the collected target scan image, a target feature vector is extracted in the same manner as feature collection of paper feature images of each era, and the target feature vector is matched with a material fingerprint recognition feature map. When the cosine similarity between the target feature vector and one of the feature vectors of the material fingerprint recognition feature map is greater than 0.85, the match is judged to be successful, and the feature vector of the paper feature image of that era is used as the target material fingerprint.

[0008] Optionally, when the target archive text data is segmented, the document text needs to be used as training data to obtain a stop word list, and then the target archive text data is segmented using the stop word list to obtain a segmentation set; The word segmentation set is matched with the preset document word library by type: each word in the word segmentation set is divided into non-age words and age words; wherein the age words contain multiple age intervals.

[0009] Optionally, the proportion of words of each era in the word segmentation set is counted, and the characteristic era is judged: if g(i) is less than a preset proportion, the era is judged not to be a characteristic era; if g(i) is greater than or equal to a preset proportion, the era is judged to be a characteristic era; wherein g(i) represents the proportion of words of the i-th era; The feature eras are sorted in descending order according to the proportion of words in each era g(i), and the feature era with the largest ranking is taken as the target text feature.

[0010] Optionally, establish a quality assessment index for the target scanned image: Q=α×n / N; where Q is the quality assessment index, n is the number of unrecognizable characters in the target archive text data, N is the number of characters in the target scanned image, and α is the clarity weight; Then, the target text quality status is determined based on the quality evaluation index of the target scanned image: if Q is less than the preset quality value, the target text quality status is determined to be normal; otherwise, the target text quality status is determined to be abnormal.

[0011] Optionally, a text type recognition model is established based on historical archive text data; When the target text quality status is normal, the word segmentation set of the target archive text data is input into the text type recognition model to obtain the text type, and the text type is added to the target text feature.

[0012] Optionally, a variant word library is established, and the text type recognition model is corrected according to the variant word library; The target archive is classified based on the target material fingerprint and the target text feature, and then the target archive is stored according to the classification: if the age of the target material fingerprint is the same as the characteristic age in the target text feature, the target archive text data is classified and stored according to the target text feature; If the age of the target material fingerprint is different from the characteristic age in the target text feature, the target file text data will be handed over to the user for secondary identification.

[0013] According to another aspect of the present application, there is provided an artificial intelligence-based archival data classification device, comprising: A data acquisition unit, which is used to: acquire target scanned images and target archive text data; A fingerprint extraction unit is used to: construct a material fingerprint knowledge graph based on the age paper feature graph, and match the target material fingerprint based on the material fingerprint knowledge graph and the collected target scan image; The text feature extraction unit is used to: segment the target archive text data to obtain a segmentation set, then perform type matching on the segmentation set and a preset document word library, and extract the target text features according to the type matching result; A quality update unit, which is used to: determine the quality status of the target text based on the target scanned image, and collect historical archive text data, and then update the target text features based on the target text quality status determination result and the historical archive text data; The data classification storage unit is used to classify the target files based on the target material fingerprint and the target text feature, and then store the target files according to the classification.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is used to control the electronic device where the computer-readable storage medium is located to execute the artificial intelligence-based archival data classification method during runtime.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: by combining multi-spectral scanning images with multi-dimensional collection of text data, a material fingerprint knowledge graph based on preset bands is constructed to achieve accurate matching of paper ages; at the same time, stop word list segmentation, age word ratio statistics and text type recognition model are dynamically updated to form a dual verification mechanism for material and text features, which effectively improves the automation level and accuracy of archive classification; the text status is dynamically monitored through the quality assessment index, and the robustness of the model is optimized by combining variant character correction and historical data training, which significantly reduces the need for manual intervention and the risk of misjudgment, and ultimately realizes intelligent decision-making for archive classification and storage, taking into account efficiency and fault tolerance, and providing an efficient and reliable full-process solution for ancient document management. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 Schematic diagram of the process of the archival data classification method based on artificial intelligence in this embodiment.

[0018] Figure 2 Schematic diagram of the process of extracting target material fingerprint according to this embodiment.

[0019] Figure 3 Schematic diagram of the flow of the method for extracting target text features in this embodiment.

[0020] Figure 4 Schematic diagram of the flow of the target text feature updating method of this embodiment.

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this embodiment.

[0022] Figure 6 A schematic diagram of the structure of the artificial intelligence-based archival data classification device provided in this embodiment. DETAILED DESCRIPTION

[0023] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and accompanying drawings. Similar components in the accompanying drawings are represented by the same reference numerals. It should be understood by those skilled in the art that the content specifically described below is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0024] It should be noted that, although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0025] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0026] Specifically, the artificial intelligence-based archival data classification method, device and medium described in this application are applied to the classification of ancient document archives; the ancient documents described in this application include ancient paper documents; the data classification of ancient document archives in this application involves: document type classification, semantic supplementation, and document text recognition.

[0027] Based on the above application scenarios, please refer to Figure 1 As shown, it is a flow chart of the artificial intelligence-based archival data classification method provided by this application, including: Step S101, collecting a target scan image and target archive text data; the target is an ancient document, and the target archive text data is text data that introduces the target in words.

[0028] It is worth noting that the target scanning image described in the present application is a multi-spectral scanning image. More specifically, the multi-spectral scanning image includes visible light, infrared, and ultraviolet band scanning images.

[0029] Please continue reading Figure 1 As shown, the artificial intelligence-based archival data classification method also includes: Step S102, extract the target material fingerprint based on the target scan image; extract the paper feature vectors of different ages based on the preset band to construct a knowledge graph, and match the target material fingerprint in combination with cosine similarity, which effectively improves the accuracy and automation level of age judgment and reduces the reliance on manual identification.

[0030] To achieve the above extraction of target material fingerprint, please refer to Figure 2 As shown, it is a schematic diagram of the process of extracting the target material fingerprint of the present application, including: Step S201, constructing a material fingerprint knowledge graph based on the age paper feature map; the age paper feature map is an exemplary image set of paper of different ages, including paper feature images of different ages; the material fingerprint knowledge graph is a set of paper feature vectors constructed based on paper feature images of different ages, which is used to identify the age of unidentified paper materials.

[0031] Specifically, the process of constructing the material fingerprint knowledge graph in step S201 is as follows: According to the preset bands, feature collection is performed on the paper feature images of each era to obtain the feature vectors of each preset band; Then, the feature vectors of each preset band of the paper feature images of each era are used as the material fingerprint knowledge graph.

[0032] Exemplarily, the preset bands described in the present application include: an ink oxidation detection band and a paper fiber detection band, wherein the ink oxidation detection band is a 450nm band, and the paper fiber detection band is an 850nm band; the characteristic vectors of each preset band can be a three-dimensional vector, which in the present application includes trivalent iron ion conversion rate, paper fiber density, and paper fiber direction, wherein the paper fiber direction is a two-dimensional component vector; the above-mentioned characteristic vector collection method is an existing public technology, and the present application does not elaborate on it.

[0033] It is worth noting that, in this embodiment, the technicians may expand the dimension of the feature vector to achieve higher matching accuracy, such as expanding the dimension of the feature vector to 128 dimensions.

[0034] Please continue reading Figure 2 As shown, the method for extracting the target material fingerprint also includes: Step S202, matching the target material fingerprint based on the material fingerprint knowledge graph and the collected target scan image.

[0035] Specifically, based on the collected target scan image, the target feature vector is extracted in the same way as the feature collection of paper feature images of each era, and the target feature vector is matched with the material fingerprint recognition feature map. When the cosine similarity between the target feature vector and one of the feature vectors of the material fingerprint recognition feature map is greater than 0.85, the match is judged to be successful, and the feature vector of the paper feature image of that era is used as the target material fingerprint.

[0036] Please continue reading Figure 1 As shown, the artificial intelligence-based archival data classification method also includes: Step S103, extracting target text features based on target archive text data.

[0037] See also Figure 3 As shown, it is a flow chart of the method for extracting target text features of the present application, including: Step S301, segmenting the target archive text data to obtain a segmentation set, and then matching the segmentation set with a preset document vocabulary.

[0038] Specifically, in step S301, when the target archive text data is segmented, the document text is used as training data to obtain a stop word list, and then the target archive text data is segmented using the stop word list to obtain a segmentation set; The word segmentation set is matched with the preset document word library by type: each word in the word segmentation set is divided into non-age words and age words; wherein the age words contain multiple age intervals.

[0039] Specifically, the era interval described in the present application is 50 years as one era interval, and history is divided into multiple era intervals; the process of "matching the word segmentation set with the preset document vocabulary" in the present application is: using the preset document vocabulary to divide each word in the word segmentation set into non-era words and era words; at the same time, the implementation process of "using document text as training data to obtain a stop word list" in the present application can be achieved by inputting the "Siku Quanshu" into the Python library to realize the training of document text; the preset document vocabulary includes: a non-era word vocabulary and a vocabulary of words for each era; the stop word list and the preset document vocabulary are used to segment the text and distinguish era words from non-era words, thereby enhancing the efficiency of text feature extraction and laying the foundation for subsequent era ratio statistics and feature era determination.

[0040] Please continue reading Figure 3 As shown, the method for extracting target text features also includes: Step S302, extracting target text features based on the type matching result.

[0041] Specifically, in step S302 described in this application: The proportion of words from each era in the word segmentation set is counted, and the characteristic era is judged: if g(i) is less than the preset proportion, the era is judged not to be a characteristic era; if g(i) is greater than or equal to the preset proportion, the era is judged to be a characteristic era; where g(i) represents the proportion of words from the i-th era; The feature eras are sorted in descending order according to the proportion of words in each era g(i), and the feature era with the largest ranking is taken as the target text feature.

[0042] Exemplarily, the proportion of words of each era described in the present application is the proportion of a certain era word in all era word types in the era word type; the present application does not make specific restrictions on the value of the preset ratio, and technical personnel in this field can set it freely, as long as the value requirement of the preset ratio is met. In the present application, the value of the preset ratio can be set to 0.6; by counting the proportions of era words and sorting them in descending order, the era with the highest proportion is used as the target text feature, thereby achieving a precise association between text content and historical era, and improving the logic and interpretability of classification.

[0043] Step S104, judging the target text quality status based on the target scanned image, and collecting historical archive text data, and then updating the target text features based on the target text quality status judgment result and the historical archive text data.

[0044] See also Figure 4 As shown, it is a schematic diagram of the process of the target text feature updating method described in this embodiment, including: Step S401: establishing a target image quality evaluation index based on a target scanned image, and determining a target text quality status.

[0045] Specifically, the process of establishing the quality assessment index of the target scanned image is as follows: Q=α×n / N; where Q is the quality assessment index, n is the number of unrecognizable characters in the target archive text data, N is the number of characters in the target scanned image, and α is the clarity weight; Then, the target text quality status is determined based on the quality evaluation index of the target scanned image: if Q is less than the preset quality value, the target text quality status is determined to be normal; otherwise, the target text quality status is determined to be abnormal.

[0046] Specifically, the present application does not impose any specific limitation on the value of the preset quality value, and those skilled in the art can freely set it as long as the value requirement of the preset quality value is met. In the present application, the preset quality value can be set to 0.6; at the same time, the optimal value of the clarity weight α is 1.2; the text quality status is automatically judged based on the quality assessment index, combined with the dynamic adjustment of the clarity weight α, to effectively identify abnormal text and ensure the data reliability of subsequent feature updates.

[0047] See also Figure 4 As shown, the target text feature updating method also includes: Step S402, collect historical archive text data, and update the target text features based on the target text quality status judgment result and the historical archive text data; it can be understood that the method of collecting historical archive text data in this application is the same as the method of collecting the target archive text data, and the historical archive text data has a document type, and the document types in this application include: history, literature, religion and technology.

[0048] Specifically, the process of updating the target text features in step S402 is as follows: Establish a text type recognition model based on historical archive text data; When the target text quality status is normal, the word segmentation set of the target archive text data is input into the text type recognition model to obtain the text type, and the text type is added to the target text feature.

[0049] Exemplarily, the process of establishing the text type recognition model described in the present application is: performing word segmentation operations on historical archive text data of different document types, and inputting the results of word segmentation into the TextCNN model to obtain text lexicons of various types; at the same time, the process of "inputting the word segmentation set of the target archive text data into the text type recognition model to obtain the text type" is: matching the word segmentation set with the text lexicon of various types, and taking the type of text lexicon with the largest number of word matches in the word segmentation set as the text type of the target archive text data; using historical archive data to train the text type recognition model, inputting the word segmentation results of normal quality text into the model, dynamically supplementing the document type features, and enhancing the adaptability and generalization ability of the classification model.

[0050] See also Figure 4 As shown, the target text feature updating method also includes: Step S403, establishing a variant word library, and calibrating the text type recognition model based on the variant word library.

[0051] Specifically, the construction of the variant character word library described in the present application can be based on the "First Batch of Variant Character Arrangement Table" as a reference; at the same time, the process of "correcting the text type recognition model" is: unifying the variant characters and standard characters in the archival text data according to the variant character word library, and eliminating the model errors caused by variant characters; by establishing a variant character word library to unify standard characters and correcting the text type recognition model, the risk of misjudgment caused by differences in character shapes is significantly reduced, and the robustness of the model is improved.

[0052] Step S105 , classifying the target files based on the target material fingerprint and the target text feature, and then storing the target files according to the classification.

[0053] Specifically, the process of multi-level classification of target files is as follows: If the age of the target material fingerprint is the same as the characteristic age in the target text feature, the target archive text data is classified and stored according to the target text feature; If the age of the target material fingerprint is different from the characteristic age in the target text feature, the target file text data will be handed over to the user for secondary identification.

[0054] For example, when the target archive is stored according to classification in the present application, if the characteristic age in the target text feature is 1654 AD and the document type is history, the target archive text data is stored as a historical classification under 1654 AD; the material fingerprint age and the text feature age are combined to automatically classify and store or trigger manual secondary identification, thereby realizing the intelligence and fault tolerance of classification decisions and optimizing the efficiency and accuracy of archive management.

[0055] The artificial intelligence-based archival data classification device provided in the embodiments of the present application can execute the artificial intelligence-based archival data classification method described in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0056] From the hardware level, this application is to realize the function of the artificial intelligence-based archival data classification method in a computer. This application also provides a computer-readable storage medium, which is set in an electronic device provided by this application. Please refer to Figure 5 As shown, the electronic device includes: A data processing unit 1, a storage unit 2, a data communication unit 3 and a bus 4; wherein the data processing unit 1 is used in the computer to implement data processing functions, the storage unit 2 implements data storage and program storage functions in the computer, the data communication unit 3 is used to implement data reception and transmission, and the bus 4 is used to implement data transmission inside the computer.

[0057] In the present application, computer-readable storage media are tangible physical storage media that can store the above-mentioned computer programs and various types of data used in the programs; the physical storage media include but are not limited to random access memory, read-only memory, optical disks, hard disks and other existing physical storage media or combinations of media.

[0058] In this embodiment, the artificial intelligence-based archival data classification method can be implemented as an executable computer program. When the computer program is loaded into the processor, when the computer program is loaded into the memory and processed by the processor through the bus, one or more steps of the artificial intelligence-based archival data classification method can be executed.

[0059] Optionally, the processor described in this embodiment can be connected to the computer program through hardware burning or other methods to complete the data execution process of the artificial intelligence-based archival data classification method.

[0060] See also Figure 6 As shown, the present application provides an artificial intelligence-based archive data classification device, comprising: A data acquisition unit, which is used to: acquire target scanned images and target archive text data; A fingerprint extraction unit is used to: construct a material fingerprint knowledge graph based on the age paper feature graph, and match the target material fingerprint based on the material fingerprint knowledge graph and the collected target scan image; The text feature extraction unit is used to: segment the target archive text data to obtain a segmentation set, then perform type matching on the segmentation set and a preset document word library, and extract the target text features according to the type matching result; A quality update unit, which is used to: determine the quality status of the target text based on the target scanned image, and collect historical archive text data, and then update the target text features based on the target text quality status determination result and the historical archive text data; The data classification storage unit is used to classify the target files based on the target material fingerprint and the target text feature, and then store the target files according to the classification.

[0061] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. An artificial intelligence-based archival data classification method, characterized in that: include: Collect target scanned images and target archive text data; Construct a material fingerprint knowledge graph based on the paper feature graph of the era, and match the target material fingerprint based on the material fingerprint knowledge graph and the collected target scan image; Segment the target archive text data to obtain a segmented word set, then match the segmented word set with a preset document word library, and extract target text features based on the type matching results; The target text quality status is judged based on the target scanned image, and historical archive text data is collected, and then the target text features are updated based on the target text quality status judgment result and the historical archive text data.

2. The method for classifying archival data based on artificial intelligence according to claim 1, characterized in that: According to the preset bands, feature collection is performed on the paper feature images of each era to obtain the feature vectors of each preset band; Then, the feature vectors of each preset band of the paper feature images of each era are used as the material fingerprint knowledge graph.

3. The artificial intelligence-based archival data classification method according to claim 2 is characterized in that: Based on the collected target scanning image, the target feature vector is extracted in the same way as the feature collection of paper feature images of each era, and the target feature vector is matched with the material fingerprint recognition feature map. When the cosine similarity between the target feature vector and one of the feature vectors of the material fingerprint recognition feature map is greater than 0.85, the match is judged to be successful, and the feature vector of the paper feature image of that era is used as the target material fingerprint.

4. The artificial intelligence-based archival data classification method according to claim 3 is characterized in that: When the target archive text data is segmented, the document text needs to be used as training data to obtain a stop word list, and then the target archive text data is segmented with the stop word list to obtain a segmentation set; The word segmentation set is matched with the preset document word library by type: each word in the word segmentation set is divided into non-age words and age words; wherein the age words contain multiple age intervals.

5. The artificial intelligence-based archival data classification method according to claim 4 is characterized in that: Count the proportion of words from each era in the word segmentation set and judge the characteristic era: if g(i) is less than the preset proportion, the era is judged not to be a characteristic era; If g(i) is greater than or equal to the preset ratio, the era is determined to be a characteristic era; wherein g(i) represents the ratio of words in the i-th era; The feature eras are sorted in descending order according to the proportion of words in each era g(i), and the feature era with the largest ranking is taken as the target text feature.

6. The artificial intelligence-based archival data classification method according to claim 5 is characterized in that: Establish the quality assessment index of the target scan image: Q=α×n / N; where Q is the quality assessment index, n is the number of unrecognizable characters in the target archive text data, N is the number of characters in the target scanned image, and α is the clarity weight; Then, the quality status of the target text is determined according to the quality evaluation index of the target scanned image: if Q is less than the preset quality value, the quality status of the target text is determined to be normal; Otherwise, the target text quality status is determined to be abnormal.

7. The artificial intelligence-based archival data classification method according to claim 1 is characterized in that: Establish a text type recognition model based on historical archive text data; When the target text quality status is normal, the word segmentation set of the target archive text data is input into the text type recognition model to obtain the text type, and the text type is added to the target text feature.

8. The artificial intelligence-based archival data classification method according to claim 7 is characterized in that: Establish a variant word library, and calibrate the text type recognition model based on the variant word library; The target archive is classified based on the target material fingerprint and the target text feature, and then the target archive is stored according to the classification: if the age of the target material fingerprint is the same as the characteristic age in the target text feature, the target archive text data is classified and stored according to the target text feature; If the age of the target material fingerprint is different from the characteristic age in the target text feature, the target file text data will be handed over to the user for secondary identification.

9. An artificial intelligence-based archival data classification device, characterized in that: include: A data acquisition unit, which is used to: acquire target scanned images and target archive text data; A fingerprint extraction unit is used to: construct a material fingerprint knowledge graph based on the age paper feature graph, and match the target material fingerprint based on the material fingerprint knowledge graph and the collected target scan image; The text feature extraction unit is used to: segment the target archive text data to obtain a segmentation set, then perform type matching on the segmentation set and a preset document word library, and extract the target text features according to the type matching result; A quality update unit, which is used to: determine the quality status of the target text based on the target scanned image, and collect historical archive text data, and then update the target text features based on the target text quality status determination result and the historical archive text data; The data classification storage unit is used to classify the target files based on the target material fingerprint and the target text feature, and then store the target files according to the classification.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is used to control the electronic device where the computer-readable storage medium is located to execute the artificial intelligence-based archival data classification method described in any one of claims 1 to 8 during operation.

Images