Electronic archive management system based on big data

Abstract

The invention discloses an electronic archive management system based on big data, and relates to the technical field of electronic archive management. The system comprises a work order creation module, a preprocessing module, a target text area positioning module, a feature extraction module, a character recognition and conversion module and an archiving and returning management module. The method comprises the following steps: creating a paper archive digital processing work order; the paper archives are preprocessed; processing the primary digital image by using an image analysis algorithm, positioning a target area, and generating a secondary digital image; identifying text content in the secondary digital image by using a pre-trained first model and a pre-trained second model, and extracting features; performing optical character recognition processing on the positioned character area to generate a primary file; and matching the primary archives with the first preset standard and the second preset standard respectively, archiving the second standard archives matched with the second preset standard, and returning the primary archives not matched with the second preset standard to realize archiving and returning management of the archives.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic archive management, and in particular relates to an electronic archive management system based on big data. Background Art

[0002] In the big data environment, the number of archives is huge and the types are diverse. Traditional manual annotation or simple template matching methods cannot meet the needs of efficient processing. The main challenges include:

[0003] 1. Variable document layout: Different types of files may have different layout styles, such as tables, lists, italics, handwriting, etc., which increases the complexity of positioning.

[0004] 2. Differences in image quality: Scans of archives may be difficult to detect due to their age and poor scan quality (such as blur, shadows, and noise).

[0005] 3. Resource consumption: When processing large-scale data, how to design efficient algorithms to reduce computing resource consumption while ensuring positioning accuracy is a core issue.

[0006] Extract key feature information from the located text area that helps to identify and distinguish text. These features can be shape, texture, frequency domain characteristics, etc. In modern methods based on deep learning, the main challenges faced by feature extraction include:

[0007] 1. Robustness and generalization: The extracted features need to have good recognition capabilities for texts of various fonts, sizes, and styles to ensure accuracy and stability under different conditions.

[0008] 2. Efficiency and scalability: As the amount of data grows, the feature extraction process needs to be efficient and adaptable to new document types and formats. Summary of the invention

[0009] In order to overcome the shortcomings and deficiencies of the above-mentioned prior art, the following technical solutions are adopted:

[0010] The electronic archive management system based on big data includes a work order creation module, a preprocessing module, a target text area positioning module, a feature extraction module, a character recognition conversion module and an archive return management module;

[0011] Work order creation module: used to create paper archive digitization processing work orders, including basic archive information and digitization requirements; the basic archive information includes archive number, name, and storage location; digitization requirements include scanning resolution, file format, and recognition accuracy;

[0012] Preprocessing module: used to preprocess paper archives, including image stitching, rotation and de-skew, trimming, decontamination, and image quality inspection operations, to improve the accuracy of subsequent text positioning and character recognition;

[0013] Target text area positioning module: used to use image analysis algorithms including edge detection, color segmentation, shape recognition, segmentation to process primary digital images, locate target areas, and generate secondary digital images;

[0014] Feature extraction module: used to identify the text content in the secondary digital image using the pre-trained first model and the second model, extract features, and compare the feature elements of the initial digital image with the corresponding feature elements in the target digital image using a two-way feature comparison mechanism until any one of the set termination conditions 1 and 2 is reached, thereby completing the matching;

[0015] Character recognition conversion module: used to perform optical character recognition processing on the located text area, convert the text in the image into an editable text format, and generate a primary file;

[0016] Archive return management module: used to match primary archives with the first preset standard and the second preset standard respectively. On the basis of matching the first preset standard, the second standard archives matching the second preset standard are archived, and the primary archives that do not match the second preset standard are returned, thereby realizing archive and return management.

[0017] Preferably, the work order creation module calls the digital processing engine interface via HTTP and passes business parameters; after the system receives the work order data, it serializes and stores it in the cache Redis as the context data of the current processing, and synchronously stores it in the database.

[0018] As a preferred embodiment, the preprocessing module workflow is as follows:

[0019] S301, scanning and reading paper files, and converting paper documents into digital images;

[0020] S302, performing a stitching process on the multiple digital images formed by the frame scanning, smoothly blending the stitching parts during the stitching process so that there is no obvious stitching trace on the whole image after the stitching, and finally merging them into a complete image;

[0021] S303, performing rotation and restoration processing on the digital image that does not conform to the reading direction, and performing deflection correction processing on the skewed image so as to achieve visual non-skewness, and during the deflection correction processing, the image deflection does not exceed 1 degree;

[0022] S304, trimming the digital image, trimming the image at a distance of at least 2 mm to 3 mm from the outermost edge of the page to remove excess white edges, reduce the size of the image file, and save storage space;

[0023] S305, performing decontamination treatment on the impurities that affect the image quality, including black spots, black lines, black frames, black edges, etc., in the image page, following the principle of showing the original appearance of the archive during the treatment process, and retaining the original paper fading spots, water stains, stains, and binding hole marks of the archive page;

[0024] S306, performing a quality check on the processed image, including the integrity, clarity, and distortion of the image. For the digital image that does not meet the quality requirements, repeat the above processing steps S302 to S305 until a primary digital image that meets the quality requirements is generated.

[0025] As a preferred embodiment, the target text area positioning module workflow is as follows:

[0026] Use edge detection operators to detect edges of images and obtain edge images; extract edge image contours as possible boundaries of target areas; convert images from RGB color space to HSV color space; define the HSV range of target colors; use the inRange function to generate a color segmentation mask; apply the mask to the original image to extract the target color area; use Hough transform to detect linear and circular targets; use shape descriptors to identify non-linear and circular shapes; extract areas that match the target shape as text target areas; segment text target areas, combine interconnected pixels in the image into independent target areas, and generate secondary digital images for subsequent individual processing of each character.

[0027] As a preferred method, the feature extraction module workflow is as follows:

[0028] The first model and the second model which have been trained in advance are used to extract features from the primary digital image and the target digital image respectively, including two groups of feature elements of the primary digital image and two groups of feature elements of the target digital image; wherein the target digital image is a digital image which is preset for reference; the first model is used to extract structural features, including lines and shapes; the second model is used to extract statistical features, including color histograms and texture feature elements;

[0029] The two groups of characteristic elements of the primary digital image include the first characteristic elements of the primary digital image output by the first model and the second characteristic elements of the primary digital image output by the second model;

[0030] The two groups of characteristic elements of the target digital image include a first characteristic element of the target digital image output by the first model and a second characteristic element of the target digital image output by the second model;

[0031] The characteristic elements of the initial digital image are compared with the corresponding characteristic elements in the target digital image until the set termination condition is reached and the matching is completed; wherein the termination condition includes termination condition 1 and termination condition 2; termination condition 1: the continuously matched characteristic points in the characteristic elements of the initial digital image and the corresponding characteristic elements in the target digital image reach the set threshold value and the image identity standard is met; termination condition 2: the offset attempt reaches the preset maximum limit, at which time it is determined whether the images are the same based on the accumulated matching results;

[0032] The specific process is as follows:

[0033] S401, selecting a first characteristic element and a second characteristic element of a target digital image and synchronously performing pairing evaluation with a first characteristic element and a second characteristic element of a corresponding primary digital image;

[0034] S402, during the feature comparison process, the threshold for continuous successful matching is set as the nth consecutive feature point alignment to preliminarily determine whether the images are identical. If the set threshold is reached, the two images are considered to be highly similar and the pairing is completed; if the continuous matching fails, the comparison is terminated and the images are considered to be inconsistent;

[0035] Continuous matching confirmation: During the comparison process, if the feature point pairs are successfully aligned for the nth time, the identity of the primary digital image and the target digital image can be preliminarily determined; on the contrary, if the continuous matching standard cannot be met, the two are deemed inconsistent and the comparison process is terminated immediately;

[0036] S403, keeping the target digital image feature elements fixed, while gradually adjusting the comparison order of the primary digital image feature elements, by adjusting the coordinate offset, to find the best matching element within the set offset range:

[0037] The offset range is set to ±N pixels, where N is a pre-set integer used to control the search interval during feature comparison;

[0038] Initialize the offset, starting from 0, that is, the original matching state without offset;

[0039] For each feature element in the primary image, the search position of the feature point in the target image is gradually adjusted according to the set offset range of ±N pixels, the coordinates of the feature element are changed, and new matching points are found in the target image; for each pair of matching points, their element matching degree is calculated. The element matching degree is used to quantify the matching degree between each pair of matching points. The element matching degree calculation formula is as follows:

[0040] a·b=a1·b1+a2·b2+...+an·bn;

[0041] Where n is the dimension of the vector, an and bn are vectors of two elements in the n-th dimensional space;

[0042] When the element matching degree is less than the set threshold, the corresponding element in the primary image is judged to be a low-quality element, marked as an invalid element, and eliminated; when the element matching degree is greater than or equal to the set threshold, the corresponding element in the primary image is judged to be a high-quality element, marked as a valid element, and retained;

[0043] Add up the element matching degrees corresponding to all valid elements to obtain the overall matching score, which is marked as image matching degree. Image matching degree quantifies the matching degree of two images;

[0044] When the image matching degree reaches the termination condition one, the search is stopped, and the image matching degree at this time is regarded as the matching degree between the final primary image and the primary array image; when the image matching degree does not reach the termination condition one, the offset N is updated, and the above operation is repeated to calculate the new element matching degree and image matching degree until the termination condition two is reached, and the search process ends.

[0045] Mark the result corresponding to the image matching with the highest score as the best matching result, and record the best matching result and the corresponding offset;

[0046] This process is repeated for all feature elements of the primary images, and the above steps are repeated until all feature elements of the primary images have been offset adjusted. Finally, the offset configuration corresponding to the highest matching quality score is marked as the best offset: it represents the overall best matching result, and the overall matching score reflects the degree of matching between the two images.

[0047] S404, determination of termination conditions;

[0048] Set termination conditions, including termination condition 1 and termination condition 2;

[0049] Termination condition 1: The number of continuously matched feature points reaches the set threshold, which is sufficient to confirm the identity of the image;

[0050] Termination condition 2: When the offset adjustment reaches the preset maximum limit, the images are judged to be identical based on the accumulated matching results;

[0051] When termination condition 1 or termination condition 2 is met, the matching is completed.

[0052] As a preferred option, the archive return management module workflow is as follows:

[0053] S701, matching the primary file with the first preset standard, and determining whether the primary file matches the first preset standard: if it does not match the first preset standard, optimizing the primary file based on the first preset standard until it matches the first preset standard, and marking it as a first standard file;

[0054] Among them, the first preset standard is the preliminary quality and format requirements, including the basic structural integrity of the document, the correct file format, the necessary metadata integrity, and the accurate filling of basic information to ensure the basic compliance of the document;

[0055] S702, matching the first standard file with the second preset standard, and determining whether the first standard file matches the second preset standard: when matching the second preset standard, marking the first standard file as the second standard file, and entering the archiving process; when not matching the second preset standard, optimizing the first standard file based on the second preset standard until matching the second preset standard, marking it as the second standard file, and entering the archiving process;

[0056] The second preset standard is based on the first preset standard, including a security setting standard, a compatibility setting standard, and a format setting standard to ensure the applicability of the document;

[0057] S703, archiving and returning;

[0058] Archiving the second standard file;

[0059] For primary archives that are successfully converted to second standard archives, a unique identification code is assigned to them, and a conversion log is recorded, which includes original file information, conversion parameters, and conversion timestamp; then, the files are classified and stored in the storage system to ensure data security and traceability; at the same time, the index database is updated;

[0060] For primary archives that cannot be successfully converted into second standard archives, they are marked as "pending" and the reasons for failure are recorded, including unsupported format, damage, encryption, and return processing.

[0061] As a preferred option, editable text formats include TXT and Word; during the processing, the context data of the executing process is locked to ensure the consistency and integrity of the data, and the lock is released after the execution is completed.

[0062] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0063] 1. The present invention processes paper archives in an automated and digital way, which greatly reduces the human involvement and manual operation links and improves the efficiency and accuracy of archive management. The entire process from work order creation to final archiving or return is digitized and streamlined, reducing human errors and delays.

[0064] 2. The present invention effectively improves the image quality through a series of operations in the preprocessing module, including image stitching, rotation correction, edge trimming and decontamination, etc., providing clearer and more accurate input data for subsequent text positioning and character recognition, which helps to improve the accuracy and efficiency of character recognition and ensure the integrity and accuracy of the digital archive content.

[0065] 3. The present invention utilizes advanced image analysis algorithms and machine learning models (such as the first and second models in the target text area positioning module and the feature extraction module) to automatically identify and extract text content in the archives, including structured data and key information. This intelligent processing not only improves work efficiency, but also makes information retrieval and utilization more convenient; by combining two-way feature comparison, continuous matching confirmation and offset comparison optimization strategies, the accuracy and robustness of image recognition are improved; by setting clear termination conditions, the efficiency and practicality of the algorithm are ensured; finally, the system outputs the best matching results, including matching point coordinates and scores, to improve the efficiency of image processing and recognition tasks.

[0066] 4. The archive return management module of the present invention screens and classifies digital archives through the first preset standard and the second preset standard, realizing automated archiving and return processing, which helps to ensure that only archives that meet the requirements are archived, and archives that do not meet the standards are returned and corrected in a timely manner, thereby improving the standardization and rigor of archive management. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 The module diagram of the electronic archive management system based on big data of the present invention is shown;

[0069] Figure 2 A flow chart of the present invention is shown. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] See also Figure 1 As shown, the electronic archive management system based on big data of this embodiment includes a work order creation module, a preprocessing module, a target text area positioning module, a feature extraction module, a character recognition conversion module and an archive return management module.

[0073] Work order creation module: used to create paper archive digitization work orders, including basic archive information and digitization requirements. The basic archive information includes archive number, name, and storage location; digitization requirements include scanning resolution, file format, and recognition accuracy.

[0074] Preprocessing module: used to preprocess paper archives, including image stitching, rotation and deskew, trimming, decontamination, and image quality inspection operations, to improve the accuracy of subsequent text positioning and character recognition.

[0075] Target text area positioning module: used to use image analysis algorithms including edge detection, color segmentation, shape recognition, segmentation to process primary digital images, locate target areas, and generate secondary digital images.

[0076] Feature extraction module: used to identify the text content in the secondary digital image and extract features using the pre-trained first model and second model. The first model is used to extract structural features, including lines and shapes; the second model is used to extract statistical features, including color histograms and texture feature elements. The feature elements of the initial digital image are compared with the corresponding feature elements in the target digital image using a two-way feature comparison mechanism until any of the set termination conditions 1 and 2 are met, and the match is completed.

[0077] Character recognition conversion module: used to perform optical character recognition processing on the located text area and convert the text in the image into an editable text format.

[0078] Archive return management module: used to match primary archives with the first preset standard and the second preset standard respectively. On the basis of matching the first preset standard, the second standard archives matching the second preset standard are archived, and the primary archives that do not match the second preset standard are returned, thereby realizing archive and return management.

[0079] The system workflow is as follows:

[0080] Step 1: Create a work order for digitizing paper archives.

[0081] The archive management system creates a paper archive digitization processing work order, which contains the basic information of the archive and the digitization requirements.

[0082] Among them, the basic information of the archive includes the archive number, name, and storage location.

[0083] Digitization requirements include scanning resolution, file format, and recognition accuracy.

[0084] Call the digital processing engine interface through API or HTTP and pass business parameters.

[0085] After receiving the work order data, the digital processing engine serializes and stores it in the cache Redis as the context data of the current processing, and synchronously stores it in the database for persistence.

[0086] Step 2: Perform image preprocessing operations.

[0087] Load the context to the currently executing image preprocessing node, and perform image stitching, rotation and deskew, trimming, decontamination, and image quality inspection operations to improve the accuracy of subsequent text positioning and character recognition.

[0088] The specific operation process is as follows:

[0089] S201, scanning and reading paper files, and converting paper documents into digital images;

[0090] S202, performing splicing processing on the multiple digital images formed by the frame scanning, smoothly blending the splicing parts during splicing, so that there is no obvious splicing trace in the whole image after splicing, and finally merging them into a complete image to ensure the integrity of the digital image of the paper archive;

[0091] S203, rotating and restoring the digital image that does not conform to the reading direction, and rectifying the skewed image so that the image does not visually feel skewed, and during the rectification process, the image skew does not exceed 1 degree;

[0092] S204, trimming the digital image, trimming the image at a distance of at least 2 mm to 3 mm from the outermost edge of the page to remove excess white edges, reduce the size of the image file, and save storage space;

[0093] S205, performing decontamination treatment on the impurities that affect the image quality, including black spots, black lines, black frames, black edges, etc., appearing on the image page, and following the principle of showing the original appearance of the archive during the treatment process, and retaining the original traces of paper fading spots, water stains, stains, binding holes, etc. on the archive page;

[0094] S206, perform quality inspection on the processed image, including image integrity, clarity, and distortion. For digital images that do not meet the quality requirements, repeat the above processing steps S302 to S305 until a primary digital image that meets the quality requirements is generated.

[0095] Step 3: Locate the target text area.

[0096] S301, using image analysis algorithms including edge detection, color segmentation, and shape recognition to segment and process primary digital images;

[0097] S302, edge detection: using an edge detection operator to perform edge detection on the image to obtain an edge image, and extracting the edge image contour as the boundary of the target area;

[0098] S303, color segmentation: convert the image from RGB color space to HSV color space; define the HSV range of the target color; use the inRange function to generate a color segmentation mask; apply the mask to the original image to extract the target color area;

[0099] S304, shape recognition: for straight and circular targets, Hough transform is used for detection; for more complex shapes, shape descriptors are used for recognition; the area matching the target shape is extracted as the potential text target area;

[0100] S305, segment the potential text target area, combine the interconnected pixels in the image into independent target areas, generate a secondary digital image, so that each character can be processed separately later; serialize the secondary digital image file and store it in the cache system REDIS as temporary context data, and simultaneously store it in the database.

[0101] Step 4: Feature extraction.

[0102] The pre-trained first model and second model are used to extract features from the primary digital image and the target digital image, including two sets of feature elements of the primary digital image and two sets of feature elements of the target digital image. The target digital image is a pre-set digital image for reference; the first model is used to extract structural features, including lines and shapes; the second model is used to extract statistical features, including color histograms and texture feature elements.

[0103] The two groups of characteristic elements of the primary digital image include the first characteristic elements of the primary digital image output by the first model and the second characteristic elements of the primary digital image output by the second model;

[0104] The dual groups of characteristic elements of the target digital image include the first characteristic elements of the target digital image output by the first model and the second characteristic elements of the target digital image output by the second model.

[0105] The feature elements of the initial digital image are compared with the corresponding feature elements in the target digital image until the set termination conditions are reached and the matching is completed. The termination conditions include termination conditions 1 and 2; Termination condition 1: The feature points of the feature elements of the initial digital image and the corresponding feature elements in the target digital image continuously match the set threshold, which is sufficient to confirm the identity of the images; Termination condition 2: The offset attempt reaches the preset maximum limit, at which time the images are judged to be identical based on the accumulated matching results.

[0106] The specific process is as follows:

[0107] S401, using a bidirectional feature comparison mechanism to perform pairing, selecting two sets of feature elements of the target digital image, and synchronously performing pairing evaluation with two sets of feature elements of the primary digital image;

[0108] S402. During the feature matching process, a threshold for continuous successful matching is set, such as the nth consecutive feature point alignment, to preliminarily determine whether the images are identical. If this standard is met, the two images are considered to be highly similar and the pairing is completed; if the continuous matching fails, the matching is terminated and the images are considered inconsistent.

[0109] Continuous matching confirmation: During the comparison process, if the feature point pairs are successfully aligned for the nth time, the identity of the primary digital image and the target digital image can be preliminarily determined; on the contrary, if the continuous matching standard cannot be met, the two are deemed inconsistent and the comparison process is terminated immediately;

[0110] S403, keeping the target digital image feature elements fixed, while gradually adjusting the comparison order of the primary digital image feature elements, by adjusting the coordinate offset, to find the best matching element within the set offset range:

[0111] The offset range is set to ±N pixels, where N is a pre-set integer used to control the search interval during feature comparison;

[0112] Initialize the offset, starting from 0, that is, the original matching state without offset;

[0113] For each feature element in the primary image, the search position of the feature point in the target image is gradually adjusted according to the set offset range of ±N pixels, the coordinates of the feature element are changed, and new matching points are found in the target image; for each pair of matching points, their element matching degree is calculated. The element matching degree is used to quantify the matching degree between each pair of matching points. The element matching degree calculation formula is as follows:

[0114] a·b=a1·b1+a2·b2+...+an·bn;

[0115] Where n is the dimension of the vector, an and bn are vectors of two elements in the n-th dimensional space;

[0116] When the element matching degree is less than the set threshold, the corresponding element in the primary image is judged to be a low-quality element, marked as an invalid element, and eliminated; when the element matching degree is greater than or equal to the set threshold, the corresponding element in the primary image is judged to be a high-quality element, marked as a valid element, and retained;

[0117] Add up the element matching degrees corresponding to all valid elements to obtain the overall matching score, which is marked as image matching degree. Image matching degree quantifies the matching degree of two images;

[0118] When the image matching degree reaches the termination condition one, the search is stopped, and the image matching degree at this time is regarded as the matching degree between the final primary image and the primary array image; when the image matching degree does not reach the termination condition one, the offset N is updated, and the above operation is repeated to calculate the new element matching degree and image matching degree until the termination condition two is reached, and the search process ends.

[0119] Mark the result corresponding to the image matching with the highest score as the best matching result, and record the best matching result and the corresponding offset;

[0120] This process is repeated for all feature elements of the primary images, and the above steps are repeated until all feature elements of the primary images have been offset adjusted. Finally, the offset configuration corresponding to the highest matching quality score is marked as the best offset: it represents the overall best matching result, and the overall matching score reflects the degree of matching between the two images.

[0121] S404: Determine the termination condition.

[0122] Set termination conditions, including termination condition one and termination condition two.

[0123] Termination condition 1: The number of continuously matched feature points reaches the set threshold, which is sufficient to confirm the identity of the image;

[0124] Termination condition 2: When the offset adjustment reaches the preset maximum limit, the images are judged to be identical based on the accumulated matching results.

[0125] When termination condition 1 or termination condition 2 is met, the matching is completed.

[0126] Step 5: Character recognition conversion generates primary files.

[0127] Perform optical character recognition on the located text area, convert the text in the image into an editable text format, and mark it as a primary file.

[0128] Step 6: File archiving and return management.

[0129] S601, matching the primary file with the first preset standard, and determining whether the primary file matches the first preset standard: if it does not match the first preset standard, optimizing the primary file based on the first preset standard until it matches the first preset standard, and marking it as a first standard file;

[0130] Among them, the first preset standard is the preliminary quality and format requirements, including but not limited to the basic structural integrity of the document, the correct file format, the completeness of necessary metadata, and the accurate filling of basic information to ensure the basic compliance of the document.

[0131] S602, matching the first standard file with the second preset standard, and determining whether the first standard file matches the second preset standard: when it matches the second preset standard, marking the first standard file as the second standard file, and entering the archiving process; when it does not match the second preset standard, optimizing the first standard file based on the second preset standard until it matches the second preset standard, marking it as the second standard file, and entering the archiving process.

[0132] Among them, the second preset standard is based on the first preset standard, including security setting standards, compatibility setting standards, and format setting standards to ensure the applicability of the document.

[0133] Ensure the quality and suitability of the document. Meeting the second preset standard means that the document is not only qualified at a basic level, but also meets more stringent professional use or regulatory compliance needs.

[0134] S603, archive and return.

[0135] Archiving the second standard file;

[0136] For primary archives that are successfully converted to second standard archives, a unique identification code is assigned to them, and a conversion log is recorded, which includes original file information, conversion parameters, and conversion timestamp; then, the files are classified and stored in the storage system to ensure data security and traceability; at the same time, the index database is updated;

[0137] For primary archives that cannot be successfully converted into second standard archives, they are marked as "pending" and the reasons for failure are recorded, including unsupported format, damage, encryption, and return processing.

[0138] The beneficial effects of this embodiment are as follows: This embodiment not only improves the efficiency and accuracy of electronic archive management by integrating advanced image processing, machine learning technology and strict quality control processes, but also promotes the effective use and long-term preservation of archive resources, providing organizations with a more reliable and efficient document management solution.

[0139] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0140] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. The electronic archive management system based on big data is characterized by: It includes work order creation module, preprocessing module, target text area positioning module, feature extraction module, character recognition conversion module and archive return management module; Work order creation module: used to create paper archive digitization processing work orders, including basic archive information and digitization requirements; the basic archive information includes archive number, name, and storage location; digitization requirements include scanning resolution, file format, and recognition accuracy; Preprocessing module: used to preprocess paper archives, including image stitching, rotation and de-skew, trimming, decontamination, and image quality inspection operations, to improve the accuracy of subsequent text positioning and character recognition; Target text area positioning module: used to use image analysis algorithms including edge detection, color segmentation, shape recognition, segmentation to process primary digital images, locate target areas, and generate secondary digital images; Feature extraction module: used to identify the text content in the secondary digital image using the pre-trained first model and the second model, extract features, and compare the feature elements of the initial digital image with the corresponding feature elements in the target digital image using a two-way feature comparison mechanism until any one of the set termination conditions 1 and 2 is reached, thereby completing the matching; Character recognition conversion module: used to perform optical character recognition processing on the located text area, convert the text in the image into an editable text format, and generate a primary file; Archive return management module: used to match primary archives with the first preset standard and the second preset standard respectively. On the basis of matching the first preset standard, the second standard archives matching the second preset standard are archived, and the primary archives that do not match the second preset standard are returned, thereby realizing archive and return management.

2. The electronic archive management system based on big data according to claim 1 is characterized in that: The work order creation module calls the digital processing engine interface through HTTP and passes business parameters; after receiving the work order data, the system serializes and stores it in the cache Redis as the context data of the current processing, and synchronously stores it in the database.

3. The electronic archive management system based on big data according to claim 1 is characterized in that: The preprocessing module workflow is as follows: S301, scanning and reading paper files, and converting paper documents into digital images; S302, performing a stitching process on the multiple digital images formed by the frame scanning, smoothly blending the stitching parts during the stitching process so that there is no obvious stitching trace on the whole image after the stitching, and finally merging them into a complete image; S303, performing rotation and restoration processing on the digital image that does not conform to the reading direction, and performing deflection correction processing on the skewed image so as to achieve visual non-skewness, and during the deflection correction processing, the image deflection does not exceed 1 degree; S304, trimming the digital image, trimming the image at a distance of at least 2 mm to 3 mm from the outermost edge of the page to remove excess white edges, reduce the size of the image file, and save storage space; S305, performing decontamination treatment on the impurities that affect the image quality, including black spots, black lines, black frames, black edges, etc., in the image page, following the principle of showing the original appearance of the archive during the treatment process, and retaining the original paper fading spots, water stains, stains, and binding hole marks of the archive page; S306, performing a quality check on the processed image, including the integrity, clarity, and distortion of the image. For the digital image that does not meet the quality requirements, repeat the above processing steps S302 to S305 until a primary digital image that meets the quality requirements is generated.

4. The electronic archive management system based on big data according to claim 1 is characterized in that: The target text area positioning module workflow is as follows: Use edge detection operators to detect edges of images and obtain edge images; extract edge image contours as possible boundaries of target areas; convert images from RGB color space to HSV color space; define the HSV range of target colors; use the inRange function to generate a color segmentation mask; apply the mask to the original image to extract the target color area; use Hough transform to detect linear and circular targets; use shape descriptors to identify non-linear and circular shapes; extract areas that match the target shape as text target areas; segment text target areas, combine interconnected pixels in the image into independent target areas, and generate secondary digital images for subsequent individual processing of each character.

5. The electronic archive management system based on big data according to claim 1 is characterized in that: The feature extraction module workflow is as follows: The first model and the second model which have been trained in advance are used to extract features from the primary digital image and the target digital image respectively, including two groups of feature elements of the primary digital image and two groups of feature elements of the target digital image; wherein the target digital image is a digital image which is preset for reference; the first model is used to extract structural features, including lines and shapes; the second model is used to extract statistical features, including color histograms and texture feature elements; The two groups of characteristic elements of the primary digital image include the first characteristic elements of the primary digital image output by the first model and the second characteristic elements of the primary digital image output by the second model; The two groups of characteristic elements of the target digital image include a first characteristic element of the target digital image output by the first model and a second characteristic element of the target digital image output by the second model; The characteristic elements of the initial digital image are compared with the corresponding characteristic elements in the target digital image until the set termination condition is reached and the matching is completed; wherein the termination condition includes termination condition 1 and termination condition 2; termination condition 1: the continuously matched characteristic points in the characteristic elements of the initial digital image and the corresponding characteristic elements in the target digital image reach the set threshold value and the image identity standard is met; termination condition 2: the offset attempt reaches the preset maximum limit, at which time it is determined whether the images are the same based on the accumulated matching results; The specific process is as follows: S401, selecting a first characteristic element and a second characteristic element of a target digital image and synchronously performing pairing evaluation with a first characteristic element and a second characteristic element of a corresponding primary digital image; S402, during the feature comparison process, the threshold for continuous successful matching is set as the nth consecutive feature point alignment to preliminarily determine whether the images are identical. If the set threshold is reached, the two images are considered to be highly similar and the pairing is completed; if the continuous matching fails, the comparison is terminated and the images are considered to be inconsistent; Continuous matching confirmation: During the comparison process, if the feature point pairs are successfully aligned for the nth time, the identity of the primary digital image and the target digital image can be preliminarily determined; on the contrary, if the continuous matching standard cannot be met, the two are deemed inconsistent and the comparison process is terminated immediately; S403, keeping the target digital image feature elements fixed, while gradually adjusting the comparison order of the primary digital image feature elements, by adjusting the coordinate offset, to find the best matching element within the set offset range: The offset range is set to ±N pixels, where N is a pre-set integer used to control the search interval during feature comparison; Initialize the offset, starting from 0, that is, the original matching state without offset; For each feature element in the primary image, the search position of the feature point in the target image is gradually adjusted according to the set offset range of ±N pixels, the coordinates of the feature element are changed, and new matching points are found in the target image; for each pair of matching points, their element matching degree is calculated. The element matching degree is used to quantify the matching degree between each pair of matching points. The element matching degree calculation formula is as follows: a·b=a1·b1+a2·b2+...+an·bn; Where n is the dimension of the vector, an and bn are vectors of two elements in the n-th dimensional space; When the element matching degree is less than the set threshold, the corresponding element in the primary image is judged to be a low-quality element, marked as an invalid element, and eliminated; when the element matching degree is greater than or equal to the set threshold, the corresponding element in the primary image is judged to be a high-quality element, marked as a valid element, and retained; Add up the element matching degrees corresponding to all valid elements to obtain the overall matching score, which is marked as image matching degree. Image matching degree quantifies the matching degree of two images; When the image matching degree reaches the termination condition 1, the search is stopped, and the image matching degree at this time is regarded as the matching degree between the final primary image and the primary array image; when the image matching degree does not reach the termination condition 1, the offset N is updated, and the above operation is repeated to calculate the new element matching degree and image matching degree until the termination condition 2 is reached, and the search process ends; Mark the result corresponding to the image matching with the highest score as the best matching result, and record the best matching result and the corresponding offset; This process is repeated for all feature elements of the primary images. All feature elements are traversed and the above steps are repeated until all feature elements of the primary images are offset adjusted. Finally, the offset configuration corresponding to the highest matching quality score is marked as the best offset, representing the overall best matching result. The overall matching score reflects the matching degree of the two images. S404, determination of termination conditions; Set termination conditions, including termination condition 1 and termination condition 2; Termination condition 1: The number of continuously matched feature points reaches the set threshold, which is sufficient to confirm the identity of the image; Termination condition 2: When the offset adjustment reaches the preset maximum limit, the images are judged to be identical based on the accumulated matching results; When termination condition 1 or termination condition 2 is met, the matching is completed.

6. The electronic archive management system based on big data according to claim 1 is characterized in that: The workflow of the archive return management module is as follows: S701, matching the primary profile with a first preset standard, and determining whether the primary profile matches the first preset standard: if the primary profile does not match the first preset standard, optimizing the primary profile based on the first preset standard until the primary profile matches the first preset standard, and marking the primary profile as a first standard profile; Among them, the first preset standard is the preliminary quality and format requirements, including the basic structural integrity of the document, the correct file format, the necessary metadata completeness, and the accurate filling of basic information to ensure the basic compliance of the document; S702, matching the first standard file with the second preset standard, and determining whether the first standard file matches the second preset standard: when matching the second preset standard, marking the first standard file as the second standard file, and entering the archiving process; when not matching the second preset standard, optimizing the first standard file based on the second preset standard until matching the second preset standard, marking it as the second standard file, and entering the archiving process; The second preset standard is based on the first preset standard, including a security setting standard, a compatibility setting standard, and a format setting standard to ensure the applicability of the document; S703, archiving and returning; Archiving the second standard file; For primary archives that are successfully converted to second standard archives, a unique identification code is assigned to them, and a conversion log is recorded, which includes original file information, conversion parameters, and conversion timestamp; then, the files are classified and stored in the storage system to ensure data security and traceability; at the same time, the index database is updated; For primary archives that cannot be successfully converted into second standard archives, they are marked as "pending" and the reasons for failure are recorded, including unsupported format, damage, encryption, and return processing.

7. The electronic archive management system based on big data according to claim 1 is characterized in that: The editable text formats include TXT and Word. During the processing, the context data of the executing process is locked to ensure the consistency and integrity of the data, and the lock is released after the execution is completed.

Images