A method and system for splicing and restoring shredded paper based on semantic error correction and association

Abstract

The present application is a method and system for splicing and restoring shredded paper based on semantic error correction and association, comprising: S1, preprocessing shredded paper images; S2, obtaining N shredded paper images, and filling the characters of the i-th row of each shredded paper image with the maximum width; S3, vertically aligning the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image; S4, sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images, and obtaining a first restored image. The method fills the character rows of each shredded paper image with the maximum width and aligns the character rows, and then establishes a correlation matrix based on the edge features between each row of the shredded paper images, finds the two shredded paper images with the highest edge correlation, and splices them horizontally, and finally performs semantic error correction and association through machine learning, and finally obtains a repaired image.

Description

Technical Field

[0001] The present application relates to the technical field of shredded paper splicing and restoration, and in particular to a shredded paper splicing and restoration method and system based on semantic error correction and association. Background Art

[0002] Paper shreds restoration plays an important role in military intelligence acquisition, historical document restoration, document security protection, text file recovery, judicial evidence recovery, and computer image processing. Since paper shreds contain text and image information, they can be restored manually, similar to jigsaw puzzles, but the efficiency is very low.

[0003] With the advancement of digital image processing technology, people have begun to use computer vision technology to automatically splice and restore shredded paper in order to improve the efficiency of shredded paper restoration.

[0004] The basic principle of digital image processing technology is to first convert shredded paper into local digital images, and then stitch multiple local digital images into a complete digital image through the feature point information on the local digital images.

[0005] The existing paper shreds splicing and restoration methods mainly calculate the edge matching degree of each local digital image, and then splice multiple local digital images according to the edge matching degree.

[0006] In the prior art, in the computer vision detection method commonly used in the shredded paper splicing and repair technology, the specific principle is: by checking the edge contour features of each shredded paper image, the matching degree of the edge contour features of each shredded paper image is calculated.

[0007] Existing stitching and repair methods have defects. When the edge contour features of the shredded paper image are few, for example, the shredded area is a blank area; or when the edge contour features of the shredded paper image are radicals such as left-falling strokes, right-falling strokes or dots, the existing feature matching formula will have errors in the matching degree calculation, resulting in a high probability of incorrect stitching.

[0008] In order to improve the success rate of splicing the edge contours of shredded paper where blank spaces or strokes are difficult to identify, the present application provides a shredded paper splicing and restoration method based on semantic error correction and association. Summary of the invention

[0009] In order to overcome the problems existing in the related art, the first aspect of the present application provides a method for splicing and restoring shredded paper based on semantic error correction and association, comprising:

[0010] S1, shredded paper image preprocessing;

[0011] S2, obtaining N shredded paper images, and filling the characters in the i-th row of each shredded paper image with the maximum width;

[0012] S3, vertically aligning the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image;

[0013] S4. Sort the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain a first restored image; wherein N is an integer greater than or equal to 2, and i is an integer greater than or equal to 1.

[0014] In one embodiment, after sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain the first restored image, the method further includes:

[0015] S5. Determine whether the first restored image is complete according to the semantic analysis model. If so, output a second restored image. If not, output an incomplete image.

[0016] In one implementation, after outputting the incomplete image, the method further includes:

[0017] S6. Fill the incomplete image according to a semantic filling model to obtain the second restored image.

[0018] In one implementation, determining whether the first restored image is complete according to the semantic analysis model specifically includes:

[0019] S501, obtaining the first restored image and performing semantic feature extraction;

[0020] S502, establishing a sequence model according to a recurrent neural network and the first restored image;

[0021] S503, inputting the first restored image into the trained semantic analysis model to obtain a character sequence;

[0022] S504, semantic segmentation and error correction are performed on the character sequence of the first restored image, and if yes, a second restored image is output; if no, an incomplete image is output.

[0023] In one implementation, filling the incomplete image according to the semantic filling model specifically includes:

[0024] S601, training a semantic filling model;

[0025] S602, acquiring and preprocessing the incomplete image;

[0026] S603: Input the incomplete image into the trained semantic filling model to obtain a second repaired image.

[0027] In one implementation, it is characterized in that the shredded paper image preprocessing specifically includes:

[0028] Obtain N shredded paper images, and convert the N shredded paper images into grayscale images respectively.

[0029] In one implementation, the acquiring of N shredded paper images and padding characters in the i-th row of each shredded paper image with a maximum width specifically includes:

[0030] S201, obtaining N grayscale images, and performing binarization processing on image pixel matrices of the grayscale images;

[0031] S202, filling the i-th row of the grayscale image with black according to a width filling algorithm;

[0032] The step of sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain a first restored image specifically includes:

[0033] S301: Obtain the Nth and N+1th grayscale images to be matched

[0034] S302 , aligning the Nth grayscale image with the N+1th grayscale image according to the number of rows.

[0035] In one implementation, performing black filling on the i-th row of the grayscale image according to a width filling algorithm specifically includes:

[0036] Determining a formula for the width filling algorithm according to the character type of the shredded paper image;

[0037] If the character type of the shredded paper image is Chinese, then executing the Chinese width filling algorithm;

[0038] If the character type of the shredded paper image is English, an English width filling algorithm is executed.

[0039] In one implementation, after aligning the number of rows of the Nth grayscale image and the N+1th grayscale image, the method further includes:

[0040] S303: If the number of rows of the Nth grayscale image is the same as that of the N+1th grayscale image, align them using the row number alignment formula.

[0041] S304: If the number of rows of the Nth grayscale image is inconsistent with the number of rows of the N+1th grayscale image, determine whether they are aligned according to the number of rows where the upper boundary of the i-th row is located.

[0042] The second aspect of the present application provides a shredded paper splicing and restoration system based on semantic error correction and association, which is used to run the method described in the first aspect of the present application, including: an image preprocessing module, an image alignment module, an image edge correlation matching module, a semantic analysis module, and a semantic filling module;

[0043] The image preprocessing module is used to execute step S1, the image alignment module is used to execute steps S2-S3, the image edge correlation matching module is used to execute step S4, the semantic analysis module is used to execute step S5, and the semantic filling module is used to execute step S6.

[0044] The technical solution provided by this application may have the following beneficial effects:

[0045] The present application provides a method for splicing and restoring shredded paper based on semantic error correction and association. The method fills the character lines of each shredded paper image with the maximum width, so that there is a color difference between the character lines and the blank lines in the shredded paper image, that is, as long as the black lines between the shredded paper images are aligned one by one, it means that the character lines have been aligned. Finally, as long as an edge correlation matrix is ​​established based on the edge features of the i-th row between the shredded paper images, and the N shredded paper images are sorted from high to low according to the edge correlation, a complete splicing and high matching restoration image can be obtained.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0048] Figure 1 It is a schematic diagram of the first process of the method for splicing and restoring shredded paper shown in an embodiment of the present application;

[0049] Figure 2 It is a second flow chart of the method for restoring shredded paper by splicing shown in the embodiment of the present application;

[0050] Figure 3 A third flow chart of the method for restoring shredded paper by splicing is shown in the embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of a fourth process of the method for splicing and restoring shredded paper shown in an embodiment of the present application;

[0052] Figure 5It is a fifth process diagram of the method for splicing and restoring shredded paper shown in the embodiment of the present application;

[0053] Figure 6 It is a sixth flow chart of the method for splicing and restoring shredded paper shown in the embodiment of the present application;

[0054] Figure 7 A schematic diagram of splicing errors in the prior art shown in an embodiment of the present application;

[0055] Figure 8 Another schematic diagram of splicing errors in the prior art shown in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0057] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0058] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0059] Embodiment 1

[0060] In the prior art, the shredded paper splicing and repair technology usually adopts a computer vision detection method. The specific principle is: by checking the edge contour features of each shredded paper image, the matching degree of the edge contour features of each shredded paper image is calculated.

[0061] Existing stitching and repair methods have defects. When the edge contour features of the shredded paper image are few, for example, the shredded area is a blank area; or when the edge contour features of the shredded paper image are radicals such as left-falling strokes, right-falling strokes or dots, the existing feature matching formula will have errors in the matching degree calculation, resulting in a high probability of incorrect stitching.

[0062] In order to improve the matching success rate of edge contours where blank spaces or strokes are difficult to identify, an embodiment of the present application provides a method for splicing and restoring shredded paper based on semantic error correction and association.

[0063] Figure 1 This is a schematic diagram of the first process of the shredded paper splicing and restoration method shown in an embodiment of the present application.

[0064] like Figure 1 As shown, the steps include:

[0065] S1. Preprocessing of shredded paper images.

[0066] In step S1, firstly, a shredded paper image file is obtained, and the shredded paper images in the shredded paper image file are converted into grayscale images respectively.

[0067] Specifically but not limiting, the shredded paper image is uploaded by a camera or obtained by scanning with a printer. The grayscale image can be digitized by MATLAB software. In a computer storage device, the grayscale image is an image matrix. In the image matrix, the two-dimensional array corresponds to the grayscale value of each pixel of the shredded paper image.

[0068] In the grayscale image corresponding to the shredded paper image, the grayscale value of the area with fonts is high, and the grayscale value of the blank area is low. After obtaining the grayscale images of N shredded paper images, step S2 is performed.

[0069] S2. Obtain N shredded paper images, and perform maximum width padding on the i-th row of each shredded paper image.

[0070] In step S2, each of the shredded paper images is recorded with i lines of words. The i-th line of words in the shredded paper image corresponds to the submatrix of the corresponding area in its image matrix. Specifically, it is determined whether the grayscale value of the matrix element of the grayscale image exceeds the preset grayscale threshold. If it exceeds, the grayscale values ​​of all matrix elements in the same row of the matrix element are adjusted to the highest, that is, all the i-th line of words in the shredded paper image are blackened.

[0071] Next, execute step S3.

[0072] S3, vertically aligning the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image;

[0073] Specifically, the widths of the i-th row of N shredded paper images are calculated respectively. If the widths of the i-th row of one shredded paper image and the i-th row of another shredded paper image are equal, the two shredded paper images are aligned.

[0074] The N shredded paper images are traversed one by one, and the N shredded paper images are preliminarily spliced.

[0075] Next, execute step S4.

[0076] S4. Sort the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain a first restored image.

[0077] After obtaining the preliminarily arranged shredded paper images in step S3, in step S4, it is determined whether the edge features between the first shredded paper image and the remaining shredded paper images match.

[0078] Specifically, the value of the right edge pixel of the first shredded paper image in the i-th row is The probability is P(R), and the value of the right edge feature of the Nth shredded paper image in the i-th row is The probability is P(L).

[0079] P(L) and P(R) represent the probability of the right edge feature of L shredded paper and the probability of the left edge feature of R shredded paper, respectively.

[0080]

[0081]

[0082] Thus, the edge correlation degree o between the first shredded paper image and the Nth shredded paper image is calculated by using the edge correlation degree calculation formula.

[0083] Specifically and non-limitingly, the edge correlation calculation formula is as follows:

[0084]

[0085] After the first shredded paper image is spliced ​​with the next shredded paper image, step S4 is performed again on the second shredded paper image until the sorting is completed.

[0086] In the embodiment of the present application, the character lines of each of the shredded paper images are filled with the maximum width, so that there is a color difference between the character lines and the blank lines in the shredded paper images. Therefore, as long as the black lines between the shredded paper images are aligned one by one, it means that the character lines have been aligned. Finally, according to the edge features between each line of the shredded paper images and the edge features of the i-th line between the shredded paper images, an edge correlation matrix is ​​established, and the N shredded paper images are sorted from high to low according to the edge correlation, a complete splicing and high matching restoration image can be obtained.

[0087] Embodiment 2

[0088] In the existing paper shreds image splicing method, splicing errors may occur, resulting in the disorder of character order.

[0089] This is because Chinese characters have continuous strokes such as left-falling strokes, right-falling strokes, and dots. If the edge features of the shredded paper image cut out continuous strokes such as left-falling strokes, right-falling strokes, and dots, the existing edge feature matching algorithm cannot match such strokes to the highest value of 1, or even far below 1, resulting in incorrect splicing.

[0090] At the same time, the existing edge feature matching algorithms mostly use clustering methods or directly use the edge features of scraps for matching, but this may cause errors due to the small number of features at the splicing point. Figure 7 The case of incorrect character splicing is considered, that is, the similarity of two different characters is the highest.

[0091] In addition, in order to repair the above splicing errors, the existing edge feature matching algorithm will use similarity to solve the problem that the characters cannot be correctly spliced ​​even if the matching degree is the highest. Although this method can provide ideas for splicing shredded paper with the same number of lines, when considering the matching degree, it is found that there are cases where both the matching degree and the similarity reach the highest value, but the characters cannot be correctly spliced ​​and restored, such as Figure 8 shown.

[0092] Therefore, in order to determine whether the shredded paper images are incorrectly spliced, after obtaining the first restored image based on the method of Example 1, the embodiment of the present application further provides a shredded paper splicing restoration method based on semantic error correction and association.

[0093] In order to determine whether the shredded paper images are incorrectly spliced, after obtaining the first restored image based on the method of Example 1, the embodiment of the present application further provides a shredded paper splicing restoration method based on semantic error correction and association.

[0094] Figure 2 This is a second flow chart of the method for restoring shredded paper by splicing shown in an embodiment of the present application.

[0095] like Figure 2 As shown, the method for restoring shredded paper based on semantic error correction and association shown in the embodiment of the present application includes the following steps:

[0096] S1, shredded paper image preprocessing;

[0097] S2, obtaining N shredded paper images, and performing maximum width padding on the i-th row of each shredded paper image;

[0098] S3, vertically aligning the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image;

[0099] S4, sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain a first restored image;

[0100] In the embodiment of the present application, the specific methods of step S1 to step S4 are consistent with those in embodiment 1 and are not described here in detail. The difference is that after step S4, step S5 is also included.

[0101] S5. Determine whether the semantics of the first restored image are correct according to the semantic analysis model. If so, output a second restored image; if not, output an incomplete image.

[0102] Figure 3 This is a third flow chart of the method for restoring shredded paper according to an embodiment of the present application.

[0103] Specifically, Figure 3 As shown, step S5 includes:

[0104] S501, training a semantic analysis model;

[0105] S502, obtaining the first restored image, and performing semantic feature extraction;

[0106] Specifically, semantic features are extracted from the first restored image using a CNN model, wherein the semantic features are shapes, edges or textures of characters.

[0107] Furthermore, in the CNN model, before the semantic feature extraction, the first restored image is subjected to convolution processing and pooling processing.

[0108] Furthermore, a CTC decoder is added to the output layer of the semantic analysis model.

[0109] In practical applications, text may contain continuous character sequences of indefinite length. In order to deal with the recognition problem of such variable-length sequences, CTC decoder is often used. CTC decoding uses blank characters to represent the intervals and repeated characters in the sequence, and improves the recognition effect by reducing the search space of the sequence.

[0110] S503, inputting the first restored image into the trained semantic analysis model to obtain a character sequence;

[0111] In step S502, the semantic features are input into a gated recurrent unit (GRU) of a recurrent neural network, and the recurrent neural network can perform character modeling on the context information recorded in the first restored image.

[0112] In the output layer of the semantic analysis model, a fully connected layer and a softmax activation function are added. The semantic analysis model converts the output into a probability distribution for each character, and then selects the most likely character as the recognition result based on the probability.

[0113] S504, semantic segmentation and error correction are performed on the character sequence of the first restored image, and if yes, the second restored image is output; if no, the incomplete image is output.

[0114] In the embodiment of the present application, when there is disordered word order of characters in the spliced ​​shredded paper image, a semantic analysis model is used to identify areas with disordered word order, thereby further improving the efficiency of repairing the shredded paper image.

[0115] Embodiment 3

[0116] When the existing edge feature matching algorithm is faced with disordered word order of spliced ​​characters or when there are omissions in the collected shredded paper images, the only option is to manually intervene and complete the sentence based on the semantic guesses of the context.

[0117] Therefore, in order to solve the problem that when there is a damaged area in the shredded paper image, the splicing success rate of the existing shredded paper splicing and restoration method in this area is greatly reduced, the embodiment of the present application also provides a shredded paper splicing and restoration method based on semantic error correction and association.

[0118] Figure 2 This is another flow chart of a method for restoring shredded paper based on semantic error correction and association provided in an embodiment of the present application.

[0119] The method for restoring shredded paper based on semantic error correction and association shown in the embodiment of the present application includes the following steps:

[0120] S1, shredded paper image preprocessing;

[0121] S2, obtaining N shredded paper images, and performing maximum width padding on the i-th row of each shredded paper image;

[0122] S3, vertically aligning the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image;

[0123] S4, sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain a first restored image;

[0124] S5, judging whether the semantics of the first restored image are correct according to the semantic analysis model, and if so, outputting a second restored image; if not, outputting a defective image;

[0125] S6. Fill the incomplete image according to a semantic filling model to obtain the second restored image.

[0126] In the embodiment of the present application, the specific methods of step S1 to step S5 are the same as those in embodiment 1 or embodiment 2 and are not described here in detail. The difference is that after step S5, step S6 is also included.

[0127] Figure 4 It is a schematic diagram of a fourth process of the method for splicing and restoring shredded paper shown in an embodiment of the present application;

[0128] Further, such as Figure 4 As shown, step S6 includes:

[0129] S601, training a semantic filling model;

[0130] In step S601, a training data set containing complete content is provided to the semantic filling model. In the embodiment of the present application, in order to speed up the acquisition efficiency of the training data set, the training data set is obtained by fine-tuning the training data through a convolutional network and according to actual task requirements.

[0131] Specifically, the training data set is used to train the semantic completion model, and the model is evaluated on a validation set to obtain a trained semantic completion model.

[0132] Specifically, the embodiments of the present application adopt a large-scale pre-trained language model and use more training data in the fine-tuning stage to improve the performance and generalization ability of the semantic completion model.

[0133] Specifically, but not limiting, the semantic filling model may be a Chinese semantic filling model or an English semantic filling model.

[0134] In an embodiment of the present application, the trained semantic completion model is loaded into the code of the method shown in the embodiment of the present application, and the semantic completion model is selectively loaded through an NLP framework (such as the TraNsformers library), allowing users to more conveniently switch models of different languages.

[0135] S602, acquiring and preprocessing the incomplete image;

[0136] In step S602, before acquiring the incomplete image, the missing area of ​​the incomplete image is marked to ensure that the context and blank parts in the incomplete image can be correctly input into the model.

[0137] S603: Input the incomplete image into the trained semantic filling model to obtain the second repaired image.

[0138] In step S603, the incomplete image is input into the trained semantic filling model. The semantic filling model predicts the missing area and gives possible candidate words or phrases, and selects the most appropriate words or phrases from the predicted structure to fill the blank part of the file, and finally obtains the second repaired image.

[0139] In an embodiment of the present application, when there are incomplete areas in the spliced ​​shredded paper image, the incomplete areas are filled using a semantic filling model, and a restored shredded paper image is output, thereby further improving the restoration efficiency of the restorer.

[0140] Embodiment 4

[0141] A method for splicing and restoring shredded paper based on semantic error correction and association as shown in embodiments one to three.

[0142] To specifically illustrate the stitching step of obtaining the first repaired image, in Embodiment 1, Embodiment 2 or Embodiment 3,

[0143] Step S1: Obtain N shredded paper images, and convert the N shredded paper images into grayscale images respectively;

[0144] In step S1, firstly, a shredded paper image file is obtained, and the shredded paper images in the shredded paper image file are converted into grayscale images respectively.

[0145] Specifically but not limiting, the shredded paper image is uploaded by a camera or obtained by scanning with a printer. The grayscale image can be digitized by MATLAB software. In a computer storage device, the grayscale image is an image matrix. In the image matrix, the two-dimensional array corresponds to the grayscale value of each pixel of the shredded paper image.

[0146] Specifically, the value range of the elements of the image pixel matrix is ​​0-255, where 0 represents black and 255 represents white. In the grayscale image corresponding to the shredded paper image, the grayscale value of the area with fonts is high, and the grayscale value of the blank area is low. After obtaining the grayscale images of N shredded paper images, step S2 is performed.

[0147] Figure 5 This is a fifth flow chart of the method for restoring shredded paper according to an embodiment of the present application.

[0148] In step S2: obtaining N shredded paper images, filling the characters in the i-th row of each shredded paper image with the maximum width, such as Figure 5 As shown, the specific steps include:

[0149] S201, performing binarization processing on the image pixel matrix of the grayscale image according to a preset grayscale threshold;

[0150] If the element value of the image pixel matrix of the grayscale image is greater than the preset grayscale threshold, modifying the current element value of the image pixel matrix to 0;

[0151] If the element value of the image pixel matrix of the grayscale image is less than the preset grayscale threshold, the current element value of the image pixel matrix is ​​modified to 1.

[0152] In step S201, the preset grayscale threshold is 254. Each element value of the current grayscale image is traversed, and when the element value of the image pixel matrix is ​​greater than 254, the current element value is modified to 0, that is, it is displayed as white on the shredded paper image; otherwise, the current element value is modified to 1, that is, it is displayed as black on the shredded paper image.

[0153] Specifically, the formula for binarizing the grayscale image is:

[0154]

[0155] At this time, the first column of the image pixel matrix represents the left edge of the shredded paper image, and the last column represents the right edge of the shredded paper image, and the preprocessing operation on the shredded paper image is completed. Thereafter, the shredded paper image is changed from a grayscale image to a black and white image.

[0156] Before splicing the N shredded paper images, each line of characters in the shredded paper images needs to be vertically aligned in the center.

[0157] Therefore, in the embodiment of the present application, after the black-and-white image of the shredded paper image is obtained, step S202 is executed to blacken a line of characters on each of the shredded paper images.

[0158] S202, determining a formula of the width filling algorithm according to the character type of the shredded paper image;

[0159] If the character type of the shredded paper image is Chinese, a Chinese width filling algorithm is executed.

[0160] In the embodiment of the present application, since the height of each Chinese character is different, it means that the height of a Chinese character cannot represent the width of a line of characters.

[0161] For example, the last character in the i-th row of the shredded paper image on the left is "high", while the first character in the i-th row of the shredded paper image on the right is "one", and the character heights of the two are significantly inconsistent. When the existing computer vision technology aligns the rows by character height, it will mistakenly think that "high" and "one" are not in the same row.

[0162] Therefore, in the embodiment of the present application, the principle of the Chinese width filling algorithm is: since the element value corresponding to the character in the image pixel matrix of the shredded paper image is 1, and the corresponding element value of the blank part is 0.

[0163] In the embodiment of the present application, a row-by-row traversal operation is performed on the image pixel matrices of the N shredded paper images. Specifically, starting from the first row of the image pixel matrix, if an element with an element value of 1 is found in this row, then all the elements in this row of the current image pixel matrix are assigned a value of 1. Operate sequentially to the last row, and the black covering operation of this picture is completed.

[0164] In the embodiment of the present application, the existing edge feature algorithm only processes edge features, while the embodiment of the present application performs black covering on the characters in the same row to facilitate the alignment processing in the next step.

[0165] If the character type of the shredded paper image is English, the English width filling algorithm is executed.

[0166] For the shredded paper image with the character type of English, the character heights of English characters are also different. For example, the capital letter "A" and the lowercase letter "a".

[0167] In order to unify the line widths of English letters, the English width filling algorithm is adopted in the embodiment of the present application.

[0168] Specifically, in the embodiment of the present application, among the characters in the i-th row, the largest character in the current row is found, such as the character "A" above. Calculate the center point coordinates (x, y) of the largest character.

[0169] Then, in the coordinate system of the current shredded paper image, the upper boundary of the current row is The lower boundary of the current row is The line width of the largest character in the current row Fill and assign the i-th row currently with the line width of the largest character.

[0170] In the embodiment of the present application, at this time, the line height of the filled i-th row is the same as the height of the largest character in this row.

[0171] After unifying the line widths of the shredded paper images, the following steps perform the alignment operation.

[0172] Specifically, in images of shredded paper cut vertically or horizontally, because the edges of the shredded paper images are straight lines rather than tortuous lines like a puzzle, the edge features are greatly reduced. When the existing edge feature algorithm faces the situation where the cut edge may happen to be a white boundary, if it is directly stitched together, it is very likely that two unmatched images will be recognized as being successfully stitched, and the stitching error rate will be relatively high.

[0173] Therefore, in the embodiment of the present application, the N shredded paper images are first aligned, that is, the i-th row of different shredded paper images are aligned against the i-th row, and then the arrangement order is adjusted.

[0174] Specifically, after step S202, step S3 is executed.

[0175] Figure 6 This is a sixth flow chart of the method for restoring shredded paper by splicing according to an embodiment of the present application.

[0176] Specifically, in step S3, if Figure 6 Step S3 includes:

[0177] S301, obtaining the Nth grayscale image to be matched;

[0178] In the embodiment of the present application, the first grayscale image is selected manually or automatically, and the number of rows is compared with the other grayscale images one by one.

[0179] S302: Determine whether the number of rows of the Nth grayscale image is consistent with the number of rows of the N+1th grayscale image.

[0180] In the embodiment of the present application, the image element matrix of the current Nth grayscale image is The number of rows is m1. The image element matrix of the current N+1th grayscale image is The number of rows is m2.

[0181] S303 . If the number of rows of the Nth grayscale image is consistent with that of the N+1th grayscale image, alignment is performed using a row number alignment formula; that is, m1=m2.

[0182] Specifically, the row alignment formula is: is the row degree of the i-th row of the N-th grayscale image, is the row degree of the i-th row of the N+1-th grayscale image.

[0183] In the embodiment of the present application, the line width of the i-th line of characters in the N-th grayscale image is compared with the line width of the i-th line of characters in the N+1-th grayscale image, and the alignment factor w of the i-th line of characters is obtained according to the line number alignment formula.i 。

[0184] Let M be a preset alignment threshold. When w i ≥ M, it is considered that the i-th row is aligned. After comparing all rows of the two pictures in sequence, if and only if each row is aligned, it means that the two pictures are aligned.

[0185] S304. If the number of rows of the N-th grayscale image and the (N + 1)-th grayscale image is inconsistent, then determine whether they are aligned according to the row number where the upper boundary of the i-th row is located, that is, m1 ≠ m2.

[0186] Since the number of rows of text in the two pictures is different at this time, as long as there is one row of text that is aligned, it is considered that the two pictures are aligned.

[0187] Specifically, first determine the row numbers k1 and k2 where the upper boundaries of the i-th rows of the N-th grayscale image and the (N + 1)-th grayscale image are located.

[0188] In the embodiments of the present application, k1 and k2 are respectively the marked positions on the row widths of the N-th grayscale image and the (N + 1)-th grayscale image. Since the shredded paper image is an image pixel matrix, k1 and k2 respectively represent the row numbers where the upper boundaries of the current rows are located in the image pixel matrix.

[0189] Furthermore, if k1 = k2, it means that the row widths of the N-th grayscale image and the (N + 1)-th grayscale image marked at the i-th row are the same, that is, it means that the current row widths are aligned.

[0190] Furthermore, if k1 ≠ k2 and m1 > m2, then assign 0 to k1 of the i-th row where there are no characters in the (N + 1)-th grayscale image.

[0191] Furthermore, if k2 ≠ k2 and m1 < m2, then assign 0 to k1 of the i-th row where there are no characters in the N-th grayscale image.

[0192] Finally, calculate the alignment factor w of the two shredded paper images through the row number alignment formula i , if it is greater than the preset alignment threshold, it means that the row numbers of the two are aligned. Execute step S4.

[0193] A shredded paper splicing and restoration method based on semantic error correction and association shown in the embodiments of the present application can horizontally align shredded paper images with different numbers of character rows, preparing for the subsequent arrangement order steps between shredded paper images.

[0194] Embodiment Five

[0195] The embodiment of the present application provides a shredded paper splicing and restoration system based on semantic error correction and association, including: an image preprocessing module, an image alignment module, an image edge correlation matching module, a semantic analysis module, and a semantic filling module;

[0196] The image preprocessing module is used to execute step S1, the image alignment module is used to execute steps S2-S3, the image edge correlation matching module is used to execute step S4, the semantic analysis module is used to execute step S5, and the semantic filling module is used to execute step S6.

[0197] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for restoring shredded paper splicing based on semantic error correction and association, characterized in that: include: S1, shredded paper image preprocessing; S2, obtaining N shredded paper images, and filling the characters in the i-th row of each shredded paper image with the maximum width; S3, vertically aligning the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image; S4, sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain a first restored image; wherein N is an integer greater than or equal to 2, and i is an integer greater than or equal to 1; The shredded paper image preprocessing specifically includes: Obtain N shredded paper images, and convert the N shredded paper images into grayscale images respectively; The acquiring of N shredded paper images and padding characters in the i-th row of each shredded paper image with a maximum width specifically includes: S201, obtaining N grayscale images, and performing binarization processing on image pixel matrices of the grayscale images; S202, filling the i-th row of the grayscale image with black according to a width filling algorithm; The step of performing black filling on the i-th row of the grayscale image according to the width filling algorithm specifically includes: Determining a formula for the width filling algorithm according to the character type of the shredded paper image; If the character type of the shredded paper image is Chinese, then executing the Chinese width filling algorithm; If the character type of the shredded paper image is English, an English width filling algorithm is executed.

2. The method for restoring shredded paper based on semantic error correction and association according to claim 1, characterized in that: After sorting the N shredded paper images according to the edge features of the i-th row between the shredded paper images to obtain the first restored image, the method further includes: S5. Determine whether the first restored image is complete according to the semantic analysis model. If so, output a second restored image. If not, output an incomplete image.

3. The method for restoring shredded paper based on semantic error correction and association according to claim 2, characterized in that: After outputting the incomplete image, the method further includes: S6. Fill the incomplete image according to a semantic filling model to obtain the second restored image.

4. The method for restoring shredded paper based on semantic error correction and association according to claim 2, characterized in that: The determining whether the first restored image is complete according to the semantic analysis model specifically includes: S501, obtaining the first restored image and performing semantic feature extraction; S502, establishing a sequence model according to a recurrent neural network and the first restored image; S503, inputting the first restored image into the trained semantic analysis model to obtain a character sequence; S504, semantic segmentation and error correction are performed on the character sequence of the first restored image, and if yes, a second restored image is output; if no, an incomplete image is output.

5. The method for restoring shredded paper based on semantic error correction and association according to claim 3, characterized in that: Filling the incomplete image according to the semantic filling model specifically includes: S601, training a semantic filling model; S602, acquiring and preprocessing the incomplete image; S603: Input the incomplete image into the trained semantic filling model to obtain a second repaired image.

6. The method for restoring shredded paper based on semantic error correction and association according to claim 1, characterized in that: The step of vertically centering the i-th row of the shredded paper image according to the row width of the i-th row of the shredded paper image specifically includes: S301, obtaining the Nth and N+1th grayscale images to be matched; S302 , aligning the Nth grayscale image with the N+1th grayscale image according to the number of rows.

7. The method for restoring shredded paper based on semantic error correction and association according to claim 6, characterized in that: The step of aligning the Nth grayscale image with the N+1th grayscale image according to the number of rows, then includes: S303, if the number of rows of the Nth grayscale image is consistent with that of the N+1th grayscale image, aligning them using a row number alignment formula; S304: If the number of rows of the Nth grayscale image is inconsistent with the number of rows of the N+1th grayscale image, determine whether they are aligned according to the number of rows where the upper boundary of the i-th row is located.

8. A paper shreds splicing and restoration system based on semantic error correction and association, characterized in that: The method according to any one of claims 1 to 7 comprises: an image preprocessing module, an image alignment module, an image edge correlation matching module, a semantic analysis module and a semantic filling module; The image preprocessing module is used to execute step S1, the image alignment module is used to execute steps S2-S3, the image edge correlation matching module is used to execute step S4, the semantic analysis module is used to execute step S5, and the semantic filling module is used to execute step S6.

Images