Book copyright page intelligent identification and information structured extraction method and device

Abstract

The invention relates to the technical field of book digitization and copyright management, and discloses a book copyright page intelligent identification and information structured extraction method and device, and the method comprises the steps: obtaining original image data, and carrying out the preprocessing; positioning a plurality of key areas by adopting a target detection model; extracting a text character string and a recognition confidence score by adopting a sequence recognition model, executing a first feedback closed loop, and if the recognition confidence is low or the semantics is incomplete, executing region coordinate self-adaptive adjustment; performing post-processing verification based on a preset rule on the text passing the first closed-loop verification; if the post-processing verification fails, activating a second feedback closed loop, positioning potential error characters and executing constraint re-identification to correct character-level identification errors; and finally, carrying out data integration and storage on the corrected text passing all verification. According to the method, the accuracy, the robustness and the automation level of information extraction are remarkably improved by setting dual dynamic correction closed loops for different error types.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of book digitization and copyright management, and particularly relates to a book copyright page intelligent recognition and information structured extraction method and device. BACKGROUND

[0002] With the accelerating process of global book digitization, the demand for cataloging and managing massive book information is increasingly urgent. As the only page carrying the core publishing information (such as ISBN, publisher, author, edition) of a book, the accurate and rapid acquisition of the information of the book copyright page is crucial for the construction of digital libraries, copyright management, and the construction of academic citation databases. In traditional information collection, the main method is manual visual inspection and manual input, which not only consumes a large amount of manpower and is low in efficiency, but also is prone to input errors due to fatigue or negligence, and is difficult to meet the needs of large-scale book informatization.

[0003] In order to improve efficiency, some institutions have begun to try to use optical character recognition (OCR) technology to automatically recognize book copyright page images. However, the existing general OCR technology faces serious challenges when directly applied to copyright page information extraction. The layout format of the book copyright page is various and lacks uniform standards, the key information is generally scattered in different positions on the page, and is often accompanied by complex backgrounds, seal interference, or scanning, shooting-induced blur and uneven light.

[0004] The existing OCR scheme has poor recognition accuracy when processing the above complex layout and low-quality images, and these technologies lack structured processing and effectiveness verification mechanisms for the recognition results. Even if the text string is recognized, the system cannot automatically determine whether the string is complete (for example, the publisher's name is truncated) or has logical errors (for example, the check digit of the ISBN number is incorrect). The inaccuracy of this recognition process, combined with the unstructured way of subsequent information processing, results in a high misreading rate of the final copyright page information, and the difficulty of data sorting and warehousing is extremely great, and still requires a large amount of manual verification and correction. SUMMARY

[0005] The purpose of the present application is to provide a book copyright page intelligent recognition and information structured extraction method and device, which solves the problems of low recognition accuracy and poor automatic processing capability caused by the lack of differential dynamic correction mechanisms for positioning errors (such as text truncation) and recognition errors (such as character confusion) in the existing OCR technology when recognizing book copyright pages.

[0006] To achieve the above purpose, the present application is implemented by the following technical solutions: The first aspect of the present application provides a method for intelligent identification and information structured extraction of a book copyright page. The method specifically comprises: obtaining original image data, and performing image preprocessing on the original image data to obtain a preprocessed image; adopting a target detection model to analyze the preprocessed image, and locating a plurality of key regions; adopting a sequence recognition model to perform information recognition and initial structuring on the plurality of key regions, so as to extract a text string and a corresponding recognition confidence score; based on the recognition confidence score and the content integrity of the text string, performing a first feedback loop: if the recognition confidence score is lower than a preset confidence lower threshold, or the text string fails the semantic integrity check, performing adaptive adjustment of region coordinates, and returning to re-extract the text string; for the text string that passes the first feedback loop check, performing post-processing check based on a preset rule; if the post-processing check fails, activating a second feedback loop: locating one or more potential error characters in the text string, and generating an alternative character set; based on the alternative character set, performing constraint re-recognition on the key region to generate a corrected text string; finally, performing data integration and storage on the corrected text string that passes the post-processing check.

[0007] As an optional implementation, the step of performing image preprocessing comprises: performing grayscale conversion on the original image data; adopting a bilateral filter to perform noise suppression on the image after grayscale conversion, so as to smooth the noise while maintaining the edge definition; adopting a local adaptive threshold method (such as Sauvola method) to perform illumination equalization and binarization on the image after noise suppression, so as to overcome the problem of uneven illumination that may exist in the original image, and obtain the preprocessed image.

[0008] As an optional implementation, the target detection model is a YOLO model. After locating the plurality of key regions, the method further comprises: performing a non-maximum suppression algorithm, and according to a preset intersection over union (IoU) threshold, filtering out redundant prediction boxes for the same target output by the target detection model.

[0009] As an optional implementation, the sequence recognition model is a model based on a convolutional recurrent neural network (CRNN). The model includes, in structure: a convolutional neural network (CNN) part for extracting deep features of an image; a recurrent neural network (RNN) part for processing sequence context information and capturing timing dependency, which can specifically adopt a bidirectional long short-term memory network (Bi-LSTM); a connectionist temporal classification (CTC) transcription portion for decoding the contextual information into the text string.

[0010] As an optional implementation, the semantic integrity check in the first feedback loop comprises: checking whether the extracted text string ends with any suffix in a predefined suffix dictionary corresponding to the category label of the key region, the predefined suffix dictionary containing integrity identifying words (e.g. publisher, year) corresponding to the category label, according to the category label (e.g. publisher, publication date) of the key region.

[0011] As an optional implementation, the region coordinate adaptive adjustment in the first feedback loop comprises: if the trigger cause is semantic incompleteness (i.e. missing suffix), expanding the width coordinate of the key region according to a preset adjustment step or adjustment ratio to generate new key region coordinates to include the end part of the text that was previously truncated.

[0012] As an optional implementation, the post-processing verification based on preset rules comprises: when the category label of the key region is ISBN, activating and executing an ISBN verification function. The verification function verifies whether the 13th digit (i.e. check digit) of the text string is the final result obtained by calculating and summing the first 12 digits according to a preset weight (e.g. alternating multiplication by 1 and 3), taking modulo 10, and then subtracting 10.

[0013] As an optional implementation, the step of activating the second feedback loop specifically comprises: if the ISBN verification fails, traversing the positions of the numeric characters in the text string, locating the potential error character causing the verification failure by replacing the characters one by one (e.g. replacing 8 with B, or replacing 0 with O) and re-executing the ISBN verification function, and generating an alternative character set containing the correct characters; the constraint re-identification adopts a CTC beam search decoding algorithm, and imposes a hard constraint in the decoding process, so that it only selects the character with the highest probability from the alternative character set at the feature sequence time step corresponding to the potential error character for decoding, instead of searching in the entire character table.

[0014] As an optional implementation, the data integration and storage comprises: extracting a unique identification field from the corrected text string that passes all verifications, the unique identification field being preferably a text string with a category label of ISBN; querying a target database with the unique identification field as an index to determine whether there is an existing data record with the same unique identification field; If no existing data record with the same unique identifier field exists in the target database, it is determined to be new data, and a data insertion operation is performed to store the corrected text string (and other key information) into the target database to achieve persistent storage.

[0015] A second aspect of the present invention provides a device for intelligent recognition and structured extraction of book copyright pages. The device includes: One or more processors; A memory that stores computer program instructions; When the computer program instructions are executed by the one or more processors, the apparatus is configured to perform any of the book copyright page intelligent recognition and information structure extraction methods as described in the first aspect of the invention.

[0016] In summary, the present invention has at least one of the following beneficial technical effects: 1. This invention achieves dynamic diagnosis and correction of recognition errors by setting up a dual feedback closed-loop correction mechanism. Specifically, the first feedback closed loop effectively solves the problem of text truncation or positioning deviation caused by inaccurate target detection and positioning through adaptive adjustment of regional coordinates; the second feedback closed loop accurately corrects character-level errors caused by misidentification by the sequence recognition model through constrained re-recognition. This phased and targeted error correction capability enables this invention to handle more complex image situations compared to the traditional one-way recognition process, significantly improving the accuracy of the final structured data.

[0017] 2. This invention achieves automatic error correction for verifiable fields by introducing post-processing verification based on preset rules (e.g., an ISBN verification function) and closely integrating it with a second feedback loop. When the system detects a logical error such as an ISBN verification failure, it can proactively activate a constrained re-identification process to locate and correct potential erroneous characters, automatically generating a high-confidence correct result, rather than simply discarding erroneous data or transferring it to a manual review queue. This reduces reliance on manual review and improves the automation level of the entire information extraction process.

[0018] 3. The technical solution of the present invention has stronger adaptability to various copyright page images in the real world. On the one hand, the image preprocessing module (e.g., using local adaptive thresholding) effectively overcomes the common problem of uneven lighting during scanning or shooting. On the other hand, the combination of semantic integrity check and adaptive adjustment of regional coordinates in the first feedback loop enables the system to actively adapt to the situation where the initial positioning of the target detection box is inaccurate due to complex layout. By dynamically expanding the search area, the system ensures complete information capture and improves the overall robustness of the system. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system hardware environment of the present invention; Figure 3 This is a functional structure diagram of the information recognition and initial structuring module of the present invention; Figure 4 This is a schematic diagram of the logic flow of the data integration and output module of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] See attached document Figure 1 This invention provides a method for intelligent identification and structured extraction of book copyright pages. This method is executed through a system containing multiple functional modules, and the collaborative work among these modules constitutes the complete technical solution of this invention.

[0022] The method first receives an original image of a book copyright page, and the image preprocessing module performs image quality optimization operations on the original image, including image denoising, contrast adjustment and binarization, to generate a preprocessed image.

[0023] Next, the key region localization module receives the preprocessed image and analyzes it to identify and locate multiple key information regions, such as regions containing publisher information, ISBN information, or author information. This module outputs a set of key regions, where each key region is defined by its bounding box coordinates and category label.

[0024] The information recognition and initial structuring module crops the corresponding key region image blocks from the preprocessed image based on the bounding box coordinates output by the key region localization module. This module performs text recognition on each key region image block to generate the corresponding text string T. k And for each text string, a recognition confidence score is calculated. This recognition confidence score S conf,k The calculation method is as follows: Where, r k For the k-th critical region, I crop (r k ) is related to r k The corresponding key region image patch, L is the text string T k The length of chari It is T k The i-th character in The symbol represents a product, from i = 1 to L, p(char i |I crop (r k )) is in a given image patch I crop (r k Under the condition that the model predicts the i-th character as char i Based on the probability, this module combines the recognized text string with the corresponding category label to form initial structured data.

[0025] Subsequently, the adaptive region adjustment module compares each recognition result output by the information recognition and the initial structuring module. The trigger condition for this comparison is Trig. A The definition is as follows: Trig A (T k ,S conf,k )=(S conf,k thresh )∨(f sem (T k ) = False); in, This is a trigger condition function that accepts two input parameters: T k and This indicates that specific conditions are selected to determine whether to trigger a certain event or process. This is the first condition: determining the current confidence level. Is it below the set threshold S? thresh S thresh It is a preset confidence threshold, (f sem (T k ) = False) This is the second condition, indicating that the function f sem (T k Whether the output of f is false. sem It is a semantic integrity check function that checks the text string T. k The function returns False if the semantics are incomplete; ∨ represents a logical OR operation. If the trigger condition Triggers... A If the condition is met, the adaptive region adjustment module adjusts the bounding box coordinates of the corresponding key region and instructs the information recognition and initial structuring module to re-execute the text recognition operation based on the adjusted new bounding box coordinates. This process constitutes the first feedback loop.

[0026] ​For structured data that has passed the first feedback closed-loop verification, the post-processing and multi-verification modules perform further validity verification. This verification includes, but is not limited to, comparison using preset industry-standard naming rules, and verification of the validity of specific fields through external interfaces or local verification algorithms. Taking ISBN-13 as an example, its verification function f... val_isbn as follows: Among them, f val,isbm (d1d2…d 13 This is a function definition indicating that it accepts a set of 13-bit numbers as input, where d1, d2, ..., d... 13 These are the individual digits of this 13-digit number, w i It is a preset weight value; when i is odd, w i =1, when i is even, w i =3, Here w i It is with each number d i The relevant weights, usually w i It is 1 or another value used for weighted summation. This summation, from i=1 to 12, represents a weighted calculation of the first 12 digits. The mod 10 operation is used to obtain the value of the last digit (units digit) of the sum. == d13 means to combine the result of the above calculation with the 13th digit d. 13 Compare them.

[0027] If the post-processing and multi-verification module fails to verify the text, the constraint re-identification module is activated. This module first analyzes and locates one or more potential error characters and their alternative character sets within the text string that caused the verification failure, based on the failed verification rules. Then, the module instructs the information recognition and initial structuring module to perform a constraint re-identification operation on the key region image block containing the potential error characters; that is, it applies constraints to the output of the potential error character positions during the decoding process. The corrected recognition result is then verified again by the post-processing and multi-verification module. This process constitutes a second feedback loop.

[0028] Finally, all structured data that has passed all the above verification steps is integrated and deduplicated by the data integration and output module. This module combines all key-value pairs into a complete data record, and after confirming the uniqueness of the information by comparing it with the existing database, it outputs high-quality structured data, completing the entire extraction process.

[0029] See attached document Figure 2The execution of the method described in this invention begins with the image data acquisition step. This image data acquisition step is implemented through a system input interface, which is used to receive raw image data containing the copyright page content of a book from one or more external image acquisition devices.

[0030] In one specific embodiment, the external image acquisition device is a flatbed scanner or document scanner. In this implementation, the system establishes a communication connection with the scanner and controls it to perform scanning operations through a standardized driver interface, such as TWAIN or WIA (Windows Image Acquisition). The system sends instructions to the scanner containing parameters such as scanning resolution and color mode. After scanning is completed, the scanner returns the generated raw image data to the system through the interface.

[0031] In this embodiment, the raw image data generated by the scanner has the following characteristics: the image illumination is provided by a controlled light source inside the scanner, resulting in uniform light distribution; simultaneously, the physical medium of the copyright page remains parallel to the scanning sensor plane during scanning, thus minimizing geometric distortion of the image. To ensure the accuracy of subsequent text recognition, the scanning resolution is set to a predetermined value, such as 300 DPI (dots per inch) or higher, to ensure that the stroke details of even the smallest characters in the image are preserved.

[0032] In another specific embodiment, the external image acquisition device is a digital camera or a mobile terminal with an integrated camera, such as a smartphone or tablet. In this implementation, the raw image data is a digital image file, which may be in JPEG, PNG, or HEIC format. This digital image file can be provided to the system in various ways, including but not limited to: transmission from the device via a wired connection (e.g., Universal Serial Bus USB) or a wireless network connection (e.g., Wi-Fi, Bluetooth); or direct capture and acquisition of the image by an application running on the system by calling the device's camera function in real time.

[0033] Unlike image data acquired by a scanner, raw image data from such handheld devices may contain shadows or highlights caused by uneven lighting in the shooting environment, as well as perspective distortion caused by the shooting angle not being perpendicular to the plane of the copyright page. These features are considered inherent to the raw image data and will be addressed in subsequent image preprocessing steps.

[0034] Regardless of the implementation method, the input interface will eventually transmit the received raw image data, i.e., a two-dimensional or three-dimensional digital signal composed of a pixel matrix, to the image preprocessing module to perform subsequent processing steps.

[0035] After the image data acquisition step, the image preprocessing module receives the original image data and performs a series of image quality enhancement operations on it to generate a preprocessed image with well-defined features that is suitable for subsequent analysis.

[0036] The first step in the image quality enhancement operation is grayscale conversion. If the original image data is a three-channel color image (e.g., RGB format), it needs to be converted into a single-channel grayscale image. This step aims to reduce the data dimensionality of subsequent processing and unify the image representation. In a specific embodiment, this conversion is achieved by weighted averaging of the red (R), green (G), and blue (B) channel components of each pixel. The conversion function can be expressed as: I gray (x,y)=w R ·I R (x,y)+w G ·I G (x,y)+w B ·I B (x,y); Among them, I gray (x,y) is the pixel value of the output grayscale image at coordinates (x,y), I R ,I G ,I B These are the pixel values ​​of the R, G, and B channels of the corresponding coordinates in the original image, w R ,w G ,w B These are preset weighting coefficients, and they satisfy w R +w G +w B =1. For example, a set of weighting coefficients that correspond to the human eye's sensitivity to different colors can be selected.

[0037] After generating the grayscale image, the image preprocessing module then performs noise suppression to reduce random noise introduced during image acquisition, such as Gaussian noise or salt-and-pepper noise. In a preferred embodiment, a bilateral filter is used to perform this operation. This filter considers both the spatial distance and pixel value differences of its neighbors when calculating the output value of a center pixel. This characteristic allows the filter to effectively smooth noise in the image while preserving the sharpness of text stroke edges to the greatest extent possible, avoiding the edge blurring problems that may occur with traditional Gaussian filtering.

[0038] The output value of a bilateral filter can be defined by the following formula: Among them, I filtered(p) is the output value of pixel p after filtering, I gray (p) and I gray (q) represents the pixel value of grayscale at positions p and q in image I. It is a spatial Gaussian function with a standard deviation of σ. s Used to calculate spatial distance weights. It is a Gaussian function with a range and a standard deviation of σ. r W is used to calculate the pixel value difference weight. p It is the normalization factor, equal to the sum of all weights. By adjusting σ... s and σ r Two parameters control the degree of filtering. This represents a Gaussian weighting function that uses the distance ||pq|| between pixel p and its neighboring pixels q to calculate the weights, with a standard deviation of σ. s , The Gaussian function here is based on the difference in gray values ​​||I gray (p)-I gray (q)|| is weighted, and the standard deviation is σ. r .

[0039] The final step in the image quality enhancement operation is illumination equalization and binarization. This step primarily addresses uneven illumination introduced by methods such as handheld device shooting. This invention employs a local adaptive thresholding method to independently calculate the binarization threshold for each pixel in the image. In a specific embodiment, the Sauvola method is used. This method dynamically calculates the threshold based on the pixel mean and standard deviation within each pixel's neighborhood window, and its calculation formula is as follows: Where T(x,y) is the binarization threshold of the pixel at coordinate (x,y), m(x,y) and s(x,y) are the mean and standard deviation of all pixels in a w×w neighborhood window centered on the pixel, respectively, k is an adjustable positive parameter, usually between 0.2 and 0.5, and R is the dynamic range of the standard deviation, usually 128.

[0040] After calculating the local threshold T(x,y) for each pixel, the filtered grayscale image I... filtered Binarization is performed. For any pixel (x, y), its value in the final preprocessed image I is... pre Pixel value I pre (x,y) is defined as: Among them, I filtered(x, y) represents the value at pixel position (x, y) in the filtered image. Through the above steps, the system generates a binary image with a white background (pixel value of 255) and black foreground text (pixel value of 0). This preprocessed image has the characteristics of low noise level, uniform illumination, and clear character edges, and is used as standardized input to be sent to the subsequent key region localization module.

[0041] The key region localization module receives the preprocessed image I output by the image preprocessing module. pre Its function is to analyze the preprocessed image through a pre-trained deep learning object detection model in order to identify and locate multiple key areas in the image that contain preset category copyright information.

[0042] In one specific embodiment, the deep learning object detection model adopts a YOLO (You Only Look Once) based architecture. This model mainly consists of three parts: a backbone network, a neck network, and a head.

[0043] The backbone network, for example, can adopt a CSPDarknet structure, and its function is to receive the preprocessed image I. pre As input, it is subjected to multi-level convolutional operations to extract a series of feature maps at different scales.

[0044] The neck network, for example, can adopt a path aggregation network (PANet) structure. Its function is to receive the multi-scale feature map output by the backbone network and effectively combine deep semantic information with shallow detail information through bidirectional path fusion from top to bottom and bottom to top to generate a set of enhanced fused feature maps.

[0045] The detection head is designed to receive the fused feature map, predict each preset anchor point on the feature map, and output a prediction tensor containing the bounding box position, target confidence, and class probability.

[0046] During the localization operation, the model will input the preprocessed image I. pre Logically, it is divided into an S×S grid. The prediction tensor output by the model contains multiple bounding box predictions generated by each grid cell. For any one of these predicted bounding boxes, the information it contains can be represented as a set of numerical values: (b x ,b y ,b w ,b h ,s obj ,pc1 ,p c2 ,...,p cN ), where b x ,b y b is the offset of the center point of the prediction box relative to the top-left corner of its grid cell. w ,b h This is for predicting the width and height of the bounding box. obj p represents a target confidence score, indicating the probability that the predicted bounding box contains any predefined category target. c1 ,...,p cN A set of category conditional probabilities represents the probability that the predicted bounding box belongs to each of N preset categories (e.g., publisher, ISBN, author, etc.) given that the predicted bounding box contains the target.

[0047] After obtaining the original prediction tensor of the model, the key region localization module decodes it and calculates the actual coordinates (x, y, y) of each prediction box in the entire image coordinate system. k ,y k ,w k ,h k Simultaneously, the final confidence score S of the prediction box for a specific category c is calculated. k,c : S k,c =s obj,k ·p c,k ; Among them, s obj,k p is the target confidence score of the k-th prediction box. c,k It is the conditional probability that the predicted box belongs to category c.

[0048] Since the model may output multiple highly overlapping predicted boxes for the same key region, the module then performs a non-maximum suppression (NMS) algorithm to filter out redundant predicted boxes. The steps of this algorithm are as follows: First, set a confidence score threshold S. thresh_det All final confidence scores S k,c Predicted boxes below this threshold are discarded.

[0049] For a specific category, the remaining prediction boxes are sorted from highest to lowest confidence score.

[0050] Select the prediction box with the highest current confidence score and use it as the final localization result.

[0051] Calculate the Intersection Over Union (IoU) ratio (LOU) of the highest-scoring predicted bounding box with all other remaining predicted bounding boxes. If the IoU value between a remaining predicted bounding box and the current highest-scoring predicted bounding box is greater than a preset IoU threshold... thresh_nms If the remaining predicted box is considered redundant, it will be discarded.

[0052] Repeat the above process until all predicted bounding boxes have been processed. Repeat the NMS process for all preset categories.

[0053] After processing with the nonmaximum suppression algorithm, the key region localization module finally outputs a set of key regions R. Each element r in this set R... k Represents a uniquely identified key region, whose data structure is r. k =(x k ,y k ,w k ,h k ,c k ), where (x k ,y k ,w k ,h k The key region is shown in the preprocessed image I. pre The precise bounding box coordinates in c k This is the category label identified for this region. This set of key regions R will be passed to the subsequent information recognition and initial structuring module.

[0054] See attached document Figure 3 The information recognition and initial structuring module receives the key region set R output by the key region localization module and the preprocessed image I output by the image preprocessing module. pre This module's function is to target each key region r in set R. k Textual information is extracted from it, and the confidence level of the extraction process is quantified to ultimately form the initial structured data.

[0055] For any element r in the key region set R k =(x k ,y k ,w k ,h k ,c k This module first determines the bounding box coordinates (x, y, y). k ,y k ,w k ,h k From the preprocessed image I pre Extract the corresponding key region image patch I from the middle crop (rk Before being fed into the subsequent recognition model, this module processes the key region image block I. crop (r k Perform a size normalization operation, for example, scaling its height to a preset fixed value while scaling its width proportionally to meet the recognition model's requirements for input size.

[0056] The module employs a text recognition model based on a convolutional recurrent neural network (CRNN) to extract text information. This model consists of three concatenated parts: The Convolutional Neural Network (CNN) part receives size-normalized key region image patches and uses them as feature extractors. It consists of multiple stacked convolutional layers, pooling layers, and activation function layers. Its function is to learn local visual features of the image and, as the network depth increases, progressively combine these local features into more abstract features. This part ultimately outputs a two-dimensional feature map, which is then converted into a sequence of feature vectors, where each time step of the sequence corresponds to a column (or a receptive field) of the feature map in the horizontal direction.

[0057] Recurrent Neural Network (RNN) Part: This part receives the feature vector sequence output by the CNN part. In a preferred embodiment, a bidirectional Long Short-Term Memory (Bi-LSTM) network is employed. This network comprises a forward LSTM and a backward LSTM, processing the feature vector sequence in two directions: from the beginning to the end and from the end to the beginning, respectively. This bidirectional processing mechanism allows predictions at any time step to utilize complete contextual information. This part ultimately outputs a new feature vector sequence, where the output vector at each time step encodes the contextual dependencies of the entire input sequence.

[0058] The transcription part receives the feature vector sequence output by the RNN and decodes it into the final text string. This embodiment employs a Connectionist Temporal Classification (CTC) layer. The CTC layer outputs a probability distribution at each time step, representing the probability of all predefined characters (including a special whitespace character) appearing at that time step. The CTC loss function allows the model to be trained end-to-end without requiring precise character-level alignment annotations.

[0059] After obtaining the output of the transcribed portion, the module performs a decoding operation to generate the text string T. kIn one embodiment, the Best Path Decoding algorithm is employed. This algorithm selects the character with the highest probability at each time step to form an initial path, and then obtains the final text string T by removing consecutively repeated characters and all whitespace. k .

[0060] At the same time, the module calculates the relationship between the text string T and the input string T. k Associated identification confidence score S conf,k .

[0061] Finally, the information recognition and initial structuring module generates the text string T. k Its corresponding recognition confidence score S conf,k , and from input r k The obtained category label c k These data units are combined into a single data unit. After all key regions have been processed, the resulting set of data units serves as the initial structured data and is then transmitted to the subsequent adaptive region adjustment module.

[0062] The adaptive region adjustment module receives the initial set of structured data units output by the information recognition and initial structuring module. The function of this module is to verify each data unit and, if the verification fails, initiate a feedback correction loop to iteratively improve the accuracy of information extraction.

[0063] For each received text string T k , Identify confidence score S conf,k and category label c k The data unit, the module first, based on a preset trigger condition Trigger, A Judge it. Trigger if and only if the condition Trigger A When the Boolean value is true, the first feedback loop is activated.

[0064] In the triggering condition Trig A In, the S thresh This is a pre-set confidence threshold, the value of which can be set independently according to different categories of key information. The f... sem (T k ,c k ) is a semantic integrity check function, whose function is to check the semantic integrity of the class label c. k The corresponding predefined rule set is used to determine the text string T. k Is the content complete? In a specific embodiment, if the category label c kFor a publishing house, the rule set may contain a suffix dictionary, such as {publishing house, publishing company, bookstore, printing house}. The function f sem Check the text string T k Check if the dictionary entry ends with any suffix from the dictionary. If it does not end with any suffix, the function returns False, thus satisfying the triggering condition.

[0065] When the first feedback loop is activated, the system performs a region coordinate adaptive adjustment operation. This operation is based on a preset inference: the trigger condition is met because the bounding box coordinates output by the upstream key region positioning module are inaccurate; for example, the bounding box is too small, causing truncation of text information. The adjustment operation is performed by a region adjustment function g. adj Execute to generate a new, revised bounding box r′. k .

[0066] The region adjustment function g adj The specific adjustment strategy is related to the triggering reason. For example: If the triggering reason is semantic incompleteness, and the predefined rules determine that the text is truncated on the right, then the function will increase the width w of the original bounding box. k Increase the preset adjustment step size Δw, or adjust it according to a preset ratio ρ. w To extend, i.e., w′ k =w k +Δw or w′ k =w k ·(1+ρ w ).

[0067] If the triggering reason is to identify the confidence score S conf,k Below the threshold S thresh If the semantic check passes, the system can execute a global expansion strategy, such as increasing the width w of the bounding box. k and height h k All are expanded by a small scale to include more contextual information. This generates a dataset containing the new coordinates (x′). k ,y′ k ,w′ k ,h′ k The new critical region r′ k .

[0068] In generating the new key region r′ k Then, the adaptive region adjustment module adjusts the new key region r′ k The information is then transmitted back to the information recognition and initial structuring module. This module is based on the new key region r′. kRe-execute the complete text information extraction and confidence assessment process to generate a new set of (T) k ′,S′ conf,k This iterative process will continue until the newly generated recognition result no longer meets the trigger condition Trig. A Or the number of adjustments performed reaches a preset maximum iteration limit N. max_adj If the maximum number of iterations N is reached... max_adj Previously, if the recognition result passed the verification, the data unit was considered valid and was sent to the subsequent post-processing and multi-verification modules. If the verification still failed after reaching the maximum number of iterations, the data unit was marked as an adjustment failure and processed according to the system's preset strategy, for example, by placing it into a set awaiting manual review.

[0069] For data units that pass the first feedback closed-loop verification, the post-processing and multi-verification module performs a final validity verification. If the verification fails, the constraint re-identification module is activated to initiate a higher-level feedback correction loop to correct data inconsistencies caused by identification errors rather than region positioning errors.

[0070] The post-processing and multi-validation module includes one or more validation rule functions. In a specific embodiment, for a text string T with the category label ISBN... k This module calls an ISBN verification function f val_isbn .

[0071] The function f val_isbn (d1d2...d 13 The specific operation is to use the first 12 digits of the string, d1 to d2. 12 and preset weight w i (where w is odd) i =1, when i is even, w i =3) Calculate a check digit and combine it with the 13th digit d of the string. 13 Compare them.

[0072] The ISBN string is considered valid only if the expression evaluates to true. Other validation rules may include, for example, determining whether a text string categorized as publication year is a four-digit number within a reasonable range.

[0073] Triggering condition for the second feedback loop B Defined as a function that returns failure for any validation rule. If a text string T k Failed to pass its corresponding verification function f val That is, f val (Tk If ) = False, then the constraint re-identification module is activated.

[0074] Upon activation, the constraint re-identification module first executes a potential error character location and candidate set generation procedure. This procedure aims to reverse-engineer the text string T based on the failed verification rules. k The most likely location for character recognition errors. Taking ISBN verification failure as an example, the program performs the following steps: Iterate through the ISBN string T k Each character position j (from 1 to 13) in the string.

[0075] For each position j, its original recognized character d j Temporarily replace each character with one of the other characters in the preset character set (0 to 9, and X) to form a temporary string.

[0076] Using the ISBN verification function f val_isbn Perform validation on each temporary string.

[0077] If a substitution exists that makes the verification function return true, then the program determines the original character d. j The character is identified as a potential error character, its position j is designated as the suspected position, and the replacement character that would allow it to pass the check is determined as the candidate character set C. cand In this specific case, the alternative character set C cand It contains only a single element.

[0078] After identifying the suspected location j and the candidate character set C cand Subsequently, the constraint re-identification module will use the original key region image block I containing the potentially erroneous character. crop (r k ), along with the suspected location j and the alternative character set C cand The information is then transmitted again to the information recognition and initial structuring module, requesting a constrained re-recognition operation. This operation differs from conventional recognition operations in that it imposes external constraints during the decoding phase. In a specific implementation, if the CTC beam search decoding algorithm is used, during the decoding process, for the feature sequence time step corresponding to the suspect position j, the decoder is restricted to only selecting from the candidate character set C. cand The search path is expanded by selecting characters. This constraint forces the output of the decoding process to conform to the format or content required by the post-processing validation rules, while still using the model's own probability output to select the most probable candidate character. This operation ultimately generates a corrected text string T. k ″.

[0079] The constraint re-identification module does not directly accept the corrected text string T. k "As a final result. Instead, it will be T." k "The data is then retransmitted to the input of the post-processing and multi-verification module for a new round of validity verification. This process constitutes the second feedback loop. This loop will continue to iterate until the corrected string passes all verifications, or the number of iterations reaches a preset maximum re-identification count limit N." max_reco Data that passes verification will be sent to the final data integration and output module; if it still fails after reaching the limit, the data will be marked as verification failed and processed according to the system's preset strategy.

[0080] See attached document Figure 4 The data integration and output module, as a downstream link in the processing flow, receives and processes all data units output by the dual closed-loop dynamic correction module (including the adaptive region adjustment module and the constraint re-identification module). This module performs final screening, cleaning, and aggregation of the data units to generate structured data records that can be stored.

[0081] Each data unit received by the data integration and output module carries a final status flag. The status flag includes at least the following: verification passed, adjustment failed (i.e., the maximum number of iterations N for the first closed loop has been reached). max_adj (If the verification still fails) or the verification fails (i.e., the maximum number of re-identification attempts N in the second closed loop is reached), max_reco (The verification still failed).

[0082] The module first performs a data screening operation. This operation iterates through all received data units and routes them according to the status flags they carry: If a data unit is marked as valid, it is considered valid data and is passed to subsequent data aggregation operations.

[0083] If a data unit is marked as having failed adjustment or failed verification, it is considered invalid. Instead of discarding such invalid data, the system routes it to a dedicated queue for manual review.

[0084] In one specific embodiment, when a data unit is routed to the manual review queue, the module simultaneously collects auxiliary information associated with the invalid data unit for the reference of the manual reviewer. The auxiliary information includes at least: the category label c of the data unit. k The last text string generated during the iterative correction process (i.e., T′) k or T″ k), and its corresponding original key region image block I crop (r k ), and a pointer to the original image I raw The identifier.

[0085] Next, the module performs a data aggregation operation on all data units determined to be valid. This operation aims to aggregate data originating from the same original image I. raw The scattered, validated key-value pairs are combined into a single, complete structured data record. In one embodiment, this aggregation operation is performed through a key-value pair in the original image I. raw This is achieved by assigning a unique image session identifier upon receipt. The module iterates through all valid data units and groups data units with the same image session identifier into the same group.

[0086] After aggregation, the module formats each group of data units to generate the structured data record. In one specific embodiment, the structured data record is formatted as a JSON (JavaScript Object Notation) object. This JSON object uses the category tag c of the data unit as its identifier. k (For example, publisher ISBN, author) is used as the key, and its corresponding fully validated text string T is used as the key. k This is used as a value. The generated JSON object is then passed to the next processing step of the data integration and output module, namely the data deduplication and storage step.

[0087] See attached document Figure 4 The data deduplication and storage step is the last operation performed by the data integration and output module, which is performed after the data aggregation operation generates structured data records.

[0088] The data integration and output module first extracts one or more preset unique identifier fields from the structured data record (e.g., a JSON object) to uniquely identify the data record. In a specific embodiment, the unique identifier field is set to a text string with the category label "is bn", because this ISBN string has undergone strict verification by the second feedback loop and has high accuracy and uniqueness.

[0089] After obtaining the unique identifier field, the module accesses a pre-configured target database. This target database is a persistent storage system that stores the final book copyright information. The module uses the extracted unique identifier field (e.g., the ISBN string) as the query condition to perform a lookup operation in a specified data table (e.g., the book information table) of the target database.

[0090] The search operation aims to determine whether a record with the exact same unique identifier field already exists in the target database. Based on the result of this search operation, the data integration and output module executes a conditional branch operation: In the first scenario, if the search operation returns one or more matching results, it indicates that the copyright information for the book already exists in the target database. The module determines the currently pending structured data record as duplicate data. To prevent data redundancy, this structured data record will not be subject to subsequent insertion operations. In one embodiment, the module may record this duplicate event (including the unique identifier field and the corresponding original image identifier) ​​in the system log for subsequent auditing.

[0091] In the second scenario, if the search operation returns no matching results, it indicates that the copyright information for the book does not exist in the target database. The module then classifies the currently pending structured data record as new data.

[0092] For all structured data records identified as new data, the module performs a data insertion operation. This operation first converts each key-value pair in the structured data record into the format of the corresponding data field in the specified data table (e.g., a book information table) in the target database, according to a preset mapping rule.

[0093] Subsequently, the module generates a data insertion instruction conforming to the database query language specification (e.g., an SQL INSERT statement). This instruction contains all the data fields to be inserted and their corresponding values. Finally, the module executes the insertion instruction, writing the new data record into the target database and completing persistent storage. The completion of this step marks the completion of the entire process of the intelligent recognition and structured information extraction method for the book copyright page of this original image.

[0094] See attached document Figure 2 The present invention also provides a device for intelligent identification and structured extraction of book copyright pages, which is configured to perform the method for intelligent identification and structured extraction of book copyright pages described in the foregoing embodiments of the present invention.

[0095] In one specific embodiment, the device may be a general-purpose or special-purpose computer device, such as a personal computer, workstation, server, or embedded system. Figure 2 As shown, the device physically may include: one or more processors, a memory, an input / output interface, and a communication interface. These components can communicate internally and be electrically connected via one or more buses.

[0096] The processor is the computing core of the device. The processor may consist of one or more central processing units (CPUs), or may include dedicated computing units to accelerate the execution of the specific computationally intensive tasks described in the method of this invention. For example, the processor may include a graphics processing unit (GPU), a neural network processing unit (NPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). These dedicated computing units are configured to efficiently perform parallel matrix operations and convolution operations of the deep learning models (e.g., YOLO architecture or CRNN architecture) involved in the key region localization module and the information recognition and initial structuring module.

[0097] The memory is used for persistent or temporary storage of data and instructions. The memory may include non-volatile memory (e.g., read-only memory ROM, flash memory) and volatile memory (e.g., random access memory RAM).

[0098] The non-volatile memory is used to store the operating system, firmware, and computer program instructions containing all the steps of the method of the present invention.

[0099] The volatile memory (RAM) is used as the main working memory of the processor to temporarily cache instructions and data loaded from the non-volatile memory during program execution, as well as intermediate data during the caching process, such as the preprocessed image, the key region image block, the feature map, and the structured data record.

[0100] The input / output interface provides the device with a means to interact with external devices and users.

[0101] The input / output interface may include an image input interface configured to connect to an external image acquisition device (e.g., a scanner or digital camera) as described in this invention and receive the raw image data.

[0102] The input / output interface may also include a data display interface (e.g., HDMI, DisplayPort) for connecting to a display to output processing results or a manual review interface.

[0103] The input / output interface may also include a data output interface (e.g., an Ethernet interface or a database connector driver) for transmitting the final structured data record to the target database.

[0104] The memory stores the computer program instructions. When the processor reads and executes the computer program instructions stored in the memory, it implements all the steps of the book copyright page intelligent recognition and information structure extraction method of the foregoing embodiments of the present invention.

[0105] Specifically, when the processor executes the computer program instructions, the instructions functionally constitute the image preprocessing module, the key region localization module, the information recognition and initial structuring module, the adaptive region adjustment module, the post-processing and multiple verification module, the constraint re-identification module, and the data integration and output module. The processor implements the complete technical solution disclosed in this invention by executing the program code segments corresponding to each of the above modules in a predetermined logical order.

Claims

1. A method for intelligent recognition and structured extraction of book copyright pages, characterized in that, Includes the following steps: Acquire the raw image data and perform image preprocessing on the raw image data to obtain the preprocessed image; The preprocessed image was analyzed using a target detection model to locate multiple key regions; A sequence recognition model is used to perform information recognition and initial structuring on the multiple key regions to extract text strings and corresponding recognition confidence scores; Based on the recognition confidence score and the content integrity of the text string, the first feedback loop is executed: if the recognition confidence score is lower than the preset confidence lower limit threshold, or the text string fails the semantic integrity check, then the region coordinates are adaptively adjusted, and the text string is retrieved again. For text strings that pass the first feedback closed-loop verification, perform post-processing verification based on preset rules; If post-processing verification fails, the second feedback loop is activated: locate one or more potential erroneous characters in the text string and generate a candidate character set; perform constrained re-identification on key areas based on the candidate character set to generate a corrected text string; Data integration and storage are performed on the corrected text strings that have passed post-processing verification.

2. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The steps for performing image preprocessing include: Perform grayscale conversion on the original image data; A bilateral filter is used to suppress noise in the grayscale converted image; A local adaptive thresholding method is used to perform illumination equalization and binarization on the noise-suppressed image to obtain the preprocessed image.

3. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The target detection model is the YOLO model; after locating multiple key regions, the extraction method further includes: The non-maximum suppression algorithm is executed to filter out redundant predicted boxes output by the object detection model.

4. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The sequence recognition model is a model based on a convolutional recurrent neural network, and the model based on a convolutional recurrent neural network includes: The convolutional neural network part used to extract image features; The recurrent neural network part used to process contextual information specifically adopts a bidirectional long short-term memory network. Connectionist temporal classification transcription portion used to decode the context information into the text string.

5. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The steps of the semantic integrity check include: Based on the category labels of key regions, check whether the text string ends with any suffix in a predefined suffix dictionary, which contains integrity identifiers corresponding to the category labels.

6. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The steps for adaptive adjustment of the region coordinates include: If the triggering reason is semantic incompleteness, the width coordinates of the key region are expanded according to the preset adjustment step size or adjustment ratio to generate new key region coordinates.

7. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The post-processing verification steps based on preset rules include: When the category label of the critical area is ISBN, the ISBN verification function is executed to verify whether the 13th digit of the text string is the check digit calculated by the first 12 digits according to the preset weight.

8. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The step of activating the second feedback loop specifically includes: Iterate through the character positions of the text string, replace each character one by one and re-execute post-processing validation to locate potential erroneous characters and generate a candidate character set; The constrained re-identification uses the CTC beam search decoding algorithm, and applies constraints during the decoding process to ensure that at the feature sequence time step corresponding to the potential erroneous character, it selects only characters from the candidate character set for decoding.

9. The method for intelligent identification and structured information extraction of book copyright pages according to claim 1, characterized in that, The data integration and storage steps include: Extract a unique identifier field from the corrected text string, wherein the unique identifier field is a text string with the category label ISBN; Query the target database to determine if there are existing data records with the same unique identifier field; If no existing data record with the same unique identifier field exists in the target database, a data insertion operation is performed to store the corrected text string into the target database.

10. A device for intelligent recognition and structured information extraction of book copyright pages, characterized in that, include: One or more processors; A memory that stores computer program instructions; When computer program instructions are executed by one or more processors, the device performs a method for intelligent recognition and structured extraction of book copyright pages as described in any one of claims 1 to 9.

Images