A multi-model cooperative book call number recognition correction method

Abstract

The present application relates to a kind of multi-model collaborative book call number identification correction method, comprising: obtaining bookshelf image;Using first model to carry out book spine detection to bookshelf image, and obtain the right book spine image by perspective transformation conversion;Using second model to locate call number from the right book spine image, obtain call number image;Using third model to carry out text line detection to call number image, obtain text line image;Using fourth model to carry out identification to text line image, output initial identification text containing mapping symbol;According to the relationship between mapping symbol and corresponding character group, mapping symbol in initial identification text is replaced with each character in the character group corresponding to the mapping symbol, to generate candidate call number;Candidate call number is compared with standard call number in standard call number database.The various models in the method have clear division of labor, realize from book spine detection to incomplete character intelligent correction, improve the accuracy and efficiency of call number identification.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-model collaborative method for book call number recognition and correction. Background Technology

[0002] The call number is the unique identifier of a library book and is the core basis for book inventory and borrowing management. During the book inventory process, the book inventory robot uses a vision module to collect images of the book spines and identify the call numbers, enabling rapid comparison of book information with the database. However, due to the limitation of the spine width, call number labels are often partially folded to the back of the book, resulting in some missing characters in the call number images collected by the robot (such as X and Y, B and P and R, 2 and 3, etc., which cannot be distinguished due to partial obstruction).

[0003] In existing technologies, solutions for handling incomplete characters mainly fall into three categories: (1) Image restoration + OCR recognition: First, the missing area is filled in using the image restoration algorithm, and then the character is recognized. This method relies on the restoration algorithm to accurately restore the original character shape. However, when the missing area is large or the character structure is complex, the restoration process may introduce fake stroke information, which may mislead the subsequent recognition. (2) OCR recognition + fuzzy matching: The OCR model outputs a "best guess" character, and then performs fault-tolerant retrieval in the database through fuzzy matching methods such as edit distance. In this method, the OCR model still needs to make a clear choice when there is insufficient information. Once the guess is wrong, the error information will irreversibly enter the retrieval link, resulting in retrieval failure or mismatch. (3) Top-K Candidates + Reordering: The OCR outputs multiple candidate characters and their confidence scores, and reorders them using a language model or contextual information. This method still limits the decision to the character level and does not fundamentally utilize the global uniqueness constraint of the call number database.

[0004] Furthermore, a single model struggles to simultaneously handle multiple sub-tasks such as spine detection, text localization, orientation determination, and incomplete character recognition, resulting in low overall recognition efficiency and insufficient robustness.

[0005] Therefore, there is an urgent need for a solution that can avoid irreversible errors introduced during the recognition stage and efficiently and accurately retrieve the call number when characters are partially missing. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-model collaborative book call number identification and correction method that can accurately identify and efficiently retrieve call numbers when characters are missing.

[0007] The technical solution adopted in this invention is a multi-model collaborative method for identifying and correcting book call numbers, comprising the following steps: S1. Obtain a bookshelf image containing multiple book spines; S2. Use the first model to detect the spine of the bookshelf image, output the rotation bounding box of each spine, and straighten the image within each rotation bounding box through perspective transformation to obtain the straightened spine image of each spine. S3. Using the second model, locate the call number region, title region, and author information region from each of the upright spine images, and extract the call number image, title image, and author image. S4. Use the third model to perform text line detection on each of the call number images, output the text line bounding box, and capture the text line image. S5. The fourth model is used to recognize each of the text lines in the image, and the initial recognized text containing the mapping symbol is output. The fourth model is trained based on a mapping strategy, which is as follows: multiple character groups are defined, each character group consists of multiple characters that cannot be distinguished from each other due to partial missing characters; a unique mapping symbol is assigned to each character group; when labeling each sample in the training dataset of the fourth model, for characters that cannot be distinguished due to partial missing characters, the character is labeled with the mapping symbol corresponding to its character group. S6. For each of the initial identification texts, according to the correspondence between the mapping symbols and character groups, the mapping symbols in the initial identification text are replaced one by one with each character in the character group corresponding to the mapping symbol, thereby generating at least one candidate call number corresponding to the initial identification text. S7. Compare the at least one candidate call number with the standard call numbers in the standard call number database. If a unique standard call number can be matched, the call number retrieval for the spine of the book is determined to be successful; otherwise, it is marked as pending manual verification.

[0008] The beneficial effects of this invention are as follows: By employing the aforementioned multi-model collaborative book call number recognition and correction method, and through multi-model collaboration and a deferred decision strategy, it solves the problem in existing technologies where folded spine labels result in missing characters, hindering accurate recognition and retrieval. At the model collaboration level, the first model accurately detects the spines in the bookshelf image and outputs rotated bounding boxes; then, perspective transformation automatically straightens each spine region, eliminating interference from complex backgrounds and tilted perspectives; the second model accurately locates and separates the call number region from the straightened spine image, avoiding interference from book titles, authors, and other text on call number recognition; the fifth model corrects the orientation of the call number image to ensure consistent text orientation; and the third model performs text line detection on the orientation-corrected call number image, accurately segmenting the call number into independent text line images. At the level of delayed decision-making strategy, this invention differs fundamentally from the serial strategies of "repair first, then identify" or "identify first, then correct" in existing technologies. It introduces a fourth model trained based on a mapping strategy to proactively classify easily confused half-characters (such as B / P / R, 2 / 3, etc.) into specific mapping symbols, reducing information loss in the character-level recognition stage to zero. The small amount of uncertainty retained in the recognition stage is quantitatively encoded as mapping symbols and passed to the candidate expansion and database retrieval stages, utilizing the global uniqueness constraint of the standard call number database to resolve local uncertainties. This strategy fundamentally avoids the risk of irreversible errors introduced by forced decisions in the recognition stage. This method overcomes the shortcomings of traditional OCR recognition models in adapting to irregular occlusion. Each model has a clear division of labor, achieving fully controllable inference from spine detection to intelligent correction of missing characters, significantly improving the accuracy and robustness of call number recognition.

[0009] Preferably, step S2 includes the following steps: S21. Input the bookshelf image into the first model, and output the rotation bounding box parameters for each spine of the bookshelf; the rotation bounding box parameters include the center coordinates and width of the rotation bounding box. ,high and rotation angle ; S22. Calculate the coordinates of the four corner points of each of the rotated bounding box parameters. , , as well as ,in The coordinates of the top left corner point, The coordinates of the top right corner point, The coordinates of the bottom right corner point, The coordinates of the bottom left corner point; S23. Establish the correspondence between the four corner points obtained in step S22 and the four corner points of the target horizontal rectangle, solve for the perspective transformation matrix H, and straighten the image within the rotating bounding box according to the perspective transformation matrix to obtain the straightened spine image of each spine; wherein, the coordinates of the four corner points of the target horizontal rectangle are respectively , , , The height of the spine image output by perspective transformation is uniformly fixed at 1024 pixels, consistent with the input size of the first model.

[0010] Preferably, step S3 includes the following steps: S31. All the upright spine images obtained in step S2 are stitched together row by row to form a stitched image; S32. Use the second model to perform target detection on the stitched image and output rectangular bounding boxes for three types of targets: call number, book title, and author. S33. Based on the rectangular bounding boxes of the three types of targets (call number, book title, and author) output, determine the spine image to which each rectangular bounding box belongs. Calculate the coordinates of each rectangular bounding box in its corresponding spine image. Then, based on the calculated coordinates, extract the call number image, book title image, and author image from the corresponding spine image.

[0011] Preferably, step S31 specifically includes the following steps: S311, Set the set of spine images for correcting the spine as follows: , Indicates the first A spine image of a book that has been turned into a standard version. Integer and Each spine image for turning the book into a standard edition The original width is The height is ; S312, For each of the said spine-aligning images Fill with a preset color along the width direction; the width after filling is... ;in, Indicates the width of the preset fill color; S313, Set the canvas height to and width are And set the canvas background color; S314. Calculate the scaling ratio Each of the upright spine images is scaled proportionally to a height of [height missing]. Width is = This yields a scaled image of the book spine. S315, using the horizontal axis = Starting from the stitching point, the scaled spine image is sequentially stitched along... Place the axis on the canvas; if the x-coordinate of the current stitching position point appears... satisfy Then, after expanding the width of the stitched image, the currently scaled spine is placed into the expanded portion of the stitched image, and the x-coordinate of the next stitching position point is updated. , If the x-coordinate of the current splicing position point appears... satisfy Then calculate the offset. Move the stitched image from the horizontal axis of the canvas Start by placing the image at the center, output the current canvas, then reinitialize the canvas and the stitched image, and continue stitching with the current scaled spine image as the starting image for the reinitialized stitched image.

[0012] Preferably, step S33 specifically includes the following steps: S331. Obtain the coordinates of the four vertices of the bounding box of the call number rectangle output by the second model, which are respectively , , , ,in, This represents the coordinates of the top-left corner point. This represents the coordinates of the top right corner point. This represents the coordinates of the bottom right corner point. This indicates the coordinates of the lower left corner point, with all coordinate values ​​taking the upper left corner of the S31 output image as the origin. S332. Calculate the x-coordinate of the center point of the rectangular bounding box containing the call number. ; S333, Based on the x-coordinate of the center point Determine the spine image to which the rectangular bounding box of the call number belongs. If the x-coordinate of the center point Image on the spine of the book If the bounding box occupies the horizontal coordinate range in the stitched image, it is determined that the bounding box belongs to the spine image. ; S334. Subtract the spine image of the book from the x-coordinate of the bounding box of the call number rectangle. The x-coordinate of the corresponding splicing position point With offset and according to the scaling ratio Convert back to the corresponding coordinates of the spine image of the book to obtain the converted coordinates; S335. Based on the transformed coordinates, extract the call number image from the corresponding spine image. S336. Following the same method as steps S331 to S335, process the book title rectangular bounding box and the author rectangular bounding box output by the second model respectively to obtain the book title image and the author image.

[0013] Preferably, before step S4, the text direction of each call number image is detected and corrected. Specifically, the fifth model is used to determine whether the text direction of the call number image is horizontal; if it is not horizontal, the call number image is rotated to the horizontal direction.

[0014] Preferably, in step S6, the expression for the number of candidate call numbers generated for each initially identified text is: ; in, This represents the total number of mapped symbols in an initial recognized text. Indicates the first The number of characters in the character group corresponding to each mapping symbol Integer and .

[0015] Preferably, step S7 includes the following specific process: The at least one candidate call number is compared with the standard call numbers in the standard call number database; If a unique matching standard call number exists, the retrieval is considered successful and the book information is recorded. If no matching standard call number is found, it will be marked as pending manual verification. If multiple matching standard call numbers exist, the title image and author image output by the second model are identified to obtain title information and author information; the title information and author information are then matched with the book information corresponding to the multiple matching standard call numbers; if a unique matching book information is found after the auxiliary matching, the retrieval is considered successful and the book information is recorded; if no matching book information is found after the auxiliary matching or multiple matching standard call numbers still exist, the book information is marked as awaiting manual verification.

[0016] Preferably, the first model is a rotated bounding box detection model based on the YOLO11 architecture; the second model is an object detection model based on the YOLO11 architecture; the third model is a text detection model based on PPOCRv5_mobile_det; the fourth model is a text recognition model based on PPOCRv5_mobile_rec; and the fifth model is an image classification model based on the PPLCNet architecture. Attached Figure Description

[0017] Figure 1 This is a flowchart of a multi-model collaborative book call number identification and correction method according to the present invention; Figure 2 This is a schematic diagram of the bookshelf image acquired in this invention; Figure 3 This is a schematic diagram of the rotated bounding box of each spine output by the first model in this invention when performing spine detection on a bookshelf image. Figure 4 This is a schematic diagram of the upright spine image obtained by perspective transformation in this invention; Figure 5 This is a schematic diagram of the stitched image formed by stitching together all the upright spine images obtained in step S2 in this invention; Figure 6 This is a schematic diagram of the rectangular bounding boxes of three types of targets—call number, book title, and author—output by the second model in this invention when performing target detection on the stitched image. Figure 7 This is a schematic diagram of the call number image in this invention; Figure 8 This is a schematic diagram of the call number image rotated to a horizontal position using the fifth model in this invention; Figure 9 This is a schematic diagram of the text line bounding box output by using the third model to perform text line detection on the call number image in this invention; Figure 10 This is a schematic diagram of obtaining a text line image based on the text line bounding box in this invention; Figure 11 This is a schematic diagram of the initial recognized text containing mapping symbols output by the fourth model in this invention when recognizing text lines from images; Figure 12 This is a comparison chart of the edit distance accuracy of different training methods in this invention; Figure 13 This is a comparison chart showing the success rate of call number retrieval using the mapping training method of this invention and not using the mapping training method. Detailed Implementation

[0018] The invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can implement it based on the description. The scope of protection of the invention is not limited to these specific embodiments.

[0019] This invention provides a multi-model collaborative method for book call number recognition and correction. This method is deployed in code on the embedded computing platform of a book inventory robot. The robot collects images of the bookshelves, identifies the book call numbers using edge computing devices, and uploads the results to a server for cloud-based correction. The method specifically includes the following steps: Step 1, such as Figure 2 As shown, obtain a bookshelf image containing multiple book spines.

[0020] Specifically, the book inventory robot moves horizontally along the aisle on one side of the bookshelf. Equipped with an industrial camera with a resolution of 3000x4000, it continuously captures images of the bookshelf at a frame rate of 25 frames per second. Using OpenCV, it reads multiple images of the bookshelf covering the entire spine of the books, ensuring that the image clarity meets the requirements for subsequent detection before proceeding with inference.

[0021] Step 2: Use the first model to detect the spines of the bookshelf image, output the rotation bounding box of each spine, and straighten the image within each rotation bounding box through perspective transformation to obtain the straightened spine image of each spine.

[0022] Specifically, the first model employs a pre-trained YOLO-based rotated bounding box detection model, BSM-OBB (BookSpine ShelfMarker Oriented Bounding Box). Using the YOLO11n-OBB pre-trained model as a foundation, several training images are proportionally scaled to a height and width of 1024 using bilinear interpolation, their color channels converted to BGR, and after pixel normalization and image augmentation, they are input into the YOLO11n-OBB model for training, resulting in the rotated bounding box detection model BSM-OBB. All image scaling during model training utilizes bilinear interpolation.

[0023] Step two includes the following specific steps: The bookshelf image is input into a first model, which outputs the rotation bounding box parameters for each spine. These parameters include the center coordinates and width of the rotation bounding box. ,high and rotation angle In the specific implementation process, the BSM-OBB rotated bounding box detection model is loaded. The BSM-OBB model outputs a tensor of shape (batch_size, params_per_box, num_boxes), for example (1, 7, 21504). `batch_size` represents the batch size, `params_per_box` represents the parameters of a single rotated bounding box, and `num_boxes` represents the number of rotated bounding boxes. The output parameter `params_per_box` for a single rotated bounding box is (…). ),in, and The coordinates of the bounding box center are For width, For high, and These represent the confidence levels for the spine and shelf number, respectively. For the rotation angle, when For rotated bounding boxes, apply non-maximum suppression to filter out the target rotated bounding boxes to retain. For example... Figure 3 As shown in the figure, the green box represents the spine category and the red box represents the shelf number category.

[0024] Calculate the coordinates of its four corner points based on each of the rotated bounding box parameters. , , as well as ,in The coordinates of the top right corner point, The coordinates of the top left corner point, The coordinates of the bottom left corner point, These are the coordinates of the bottom right corner. Where, the coordinates... , , as well as The expression is: .

[0025] Establish the correspondence between the four corner points of the rotated bounding box obtained in step S22 and the four corner points of the target horizontal rectangle, solve for the perspective transformation matrix H, and perform a perspective transformation. The coordinates of the perspective-transformed target horizontal rectangle corresponding to the rotated bounding box are: .

[0026] Because the perspective transformation matrix H is scale-invariant in homogeneous coordinates—meaning that for any non-zero scalar k, matrices kH and H represent the same geometric transformation—matrices H are normalized before solving to eliminate this ambiguity and obtain a definite solution. Since the spine plane is not parallel to the camera's imaging plane, the non-homogeneous terms of the perspective transformation matrix... The normalization operation is always valid as the element is never zero, and it will always be applied to the bottom right element. Set to a fixed value of 1, that is, let .

[0027] This operation reduces the degrees of freedom of matrix H from 9 to 8, thus transforming the nonlinear perspective transformation equation into a linear system of equations Ah=b with respect to the remaining 8 parameters, making the problem solvable stably using standard linear algebra methods. When the four corner points form a nondegenerate quadrilateral (with no three points collinear), Ah=b can be solved directly to obtain a unique solution; that is: .

[0028] For each pixel coordinate in the output upright spine image (size w×h) Through the inverse matrix of the perspective transformation matrix The unnormalized homogeneous coordinate vector on the source image plane is calculated. : .

[0029] The unnormalized homogeneous coordinates are normalized to obtain floating-point coordinates in the rotated bounding box image coordinate system. ,in Based on the calculated floating-point coordinates The algorithm uses bilinear interpolation to directly sample pixel values ​​from the input image. The sampled pixel values ​​are then assigned to the output image. Location.

[0030] Through the above steps, a perspective transformation is performed on the rotating rectangular region in the original image, bounded by four vertices P1, P2, P3, and P4. The transformation maps the rotating rectangular region onto a horizontal rectangular canvas of size w × h, as shown below. Figure 4 As shown, a corrected, untilted target image is extracted.

[0031] Step 3: Using the second model, locate the call number area, title area, and author information area from each of the corrected spine images, and extract the call number image, title image, and author image.

[0032] The purpose of step three is to accurately locate the area containing the call number in each spine image of the book and separate it from other text information such as the title and author, and process the call number text separately.

[0033] Specifically, the second model employs the pre-trained YOLO11-based object detection model BTCNA-Det (Book Title Call Number Author Detection), whose task is to detect three types of bounding boxes: book title, call number, and author. Using the YOLO11n-Det pre-trained model as a foundation, several training images are proportionally padded and scaled to a height and width of 640 pixels, their color channels are converted to BGR, and pixel normalization and image augmentation are performed before training to obtain the object detection model BTCNA-Det.

[0034] Step three includes the following specific steps: S31. Stitch all the corrected spine images obtained in step two into at least one stitched image in the horizontal direction. Since the call number area of ​​a single book spine is small, stitch multiple corrected spine images row by row to form a 640×640 image that is adapted to the input of the BTCNA-Det model, thereby improving the model's inference efficiency.

[0035] S311, Set the set of spine images for correcting the spine as follows: , Indicates the first A spine image of a book that has been turned into a standard version. Integer and Each spine image for turning the book into a standard edition The original width is The height is .

[0036] S312. To facilitate subsequent differentiation of the boundaries between adjacent spines in the stitched images, for each of the straightened spine images... Fill the width with a preset color, i.e., fill a pixel strip. The preset color can be a solid color that contrasts sharply with the spine background color. The width after filling is... ;in, This indicates the width of the preset fill color; the height of the preset fill color remains constant. The fill color should be consistent with the fill width. The input should be consistent with the training set samples. (The padded image...) Specifically, it is expressed as follows: .

[0037] S313, Set the initial rectangular canvas Size is ,in, 3 indicates the BGR color channel, fill color. This is to ensure compatibility with the 640×640 image input for subsequent BTCNA-Det models. Set the stitching pattern. The height is Its width will change dynamically based on subsequent processing.

[0038] S314. Since the spine images may have different heights, they are all scaled to the same height as the canvas to ensure that no image exceeds the canvas boundary in the height direction. Calculate the scaling ratio. This involves taking the minimum value between the height and width ratios to ensure that the scaled image does not exceed the canvas limits in the width direction. Each of the straightened spine images is then scaled according to the specified ratio. Scale proportionally to height of Width is The scaled-up, upright spine image is obtained. .

[0039] S315, Let Indicates processing the first The x-coordinate of the current stitching position when stitching an image is performed, its value is determined by the previous iteration. Updated: initial value The scaled spine images are sequentially arranged along... Place the image along the axis on the canvas; for the currently scaled spine image to be placed, determine the x-coordinate of the current stitching position point. Does it meet the requirements? .

[0040] Scenario 1: If satisfied This indicates that there is still enough space on the canvas to accommodate the image. The following operations will then be performed: Region Expansion and Initialization: This involves updating the mosaic pattern. The width expands to the right The newly added pixel is assigned a value of 0: ; Vertical Alignment and Image Overlay: Calculating Vertical Offset The filled and scaled spine image Coverage Corresponding sub-regions: And update the x-coordinate of the next stitching position. .

[0041] Scenario 2: If not satisfied This indicates that the remaining width of the current canvas is insufficient to accommodate the image. In this case, perform the following operation: Calculate the offset. ,Will The image is placed in the center of the canvas: , Output the current canvas.

[0042] After all the spine images have been processed, the last stitched image is then processed. Output the last image using the centering method as in scenario two. .

[0043] Through the above steps, one or more processed images are obtained. Each processed image contains several spine images aligned vertically and arranged sequentially horizontally, such as... Figure 5 As shown.

[0044] S32. Load the second model and perform target detection on each stitched image. The second model, after training, can identify and locate three types of targets. The output of the second model, after non-maximum processing, is in the following format: ,in, and The coordinates of the bounding box center are The width of the bounding box. This is the height of the bounding box. These are the confidence scores for the call number bounding box, the title bounding box, and the author bounding box, respectively. For example... Figure 6 As shown, it can identify and locate: call number (category label is call_number), book title (category label is title), and author information (category label is author). In the figure, the green box is the call number category, the red box is the book title category, and the blue box is the author category.

[0045] S33. Based on the rectangular bounding boxes of the three target categories (call number, title, and author) output, determine the spine image to which each bounding box belongs. Then, based on the coordinates of each bounding box, calculate its coordinates within its corresponding spine image. Finally, based on the calculated coordinates, extract the call number image, title image, and author image from the spine image. The specific process of step S33 is as follows: S331. Obtain the coordinates of the four vertices of the bounding box of the call number rectangle output by the second model. The coordinates of the four corners of the bounding box are denoted as follows: , , , ,in, This represents the coordinates of the top-left corner. This represents the coordinates of the top right corner point. This represents the coordinates of the bottom right corner point. This represents the coordinates of the lower left corner point, with the origin set to the upper left corner of the stitched image output by S31.

[0046] S332. Calculate the x-coordinate of the center point of the rectangular bounding box of the call number. , This involves taking the average of the x-coordinates of the top-left and bottom-right corners of the call number rectangle. Using the x-coordinate of the center point instead of the boundary coordinates for attribution determination avoids ambiguity when the bounding box crosses the spine image boundary.

[0047] S333, Based on the x-coordinate of the center point Determine the scaled, upright spine image to which the bounding box of the call number belongs. If the x-coordinate of the center point is marked Falling onto the scaled-up, upright spine image Within the horizontal coordinate range occupied by the stitched image, the bounding box is determined to belong to the scaled, uprighted spine image. During the stitching process in step S315, the system has recorded the starting position of the horizontal coordinate in each stitched image. End position and offset in the canvas image According to the starting position End position and offset It can be confirmed The horizontal coordinate range occupied in the stitched image.

[0048] S334. Transform the coordinates of all vertices of the call number rectangle bounding box from the stitched image coordinate system back to the original spine-turned image coordinate system. That is: subtract the coordinates of the spine-turned image from the x-coordinate of the call number rectangle bounding box. The x-coordinate of the corresponding splicing position point With offset And the coordinates are scaled up. This is the ratio of the input size of model BSM-OBB to the input size of model BTCNA-Det, i.e. The true coordinates of the bounding box of the call number in the corresponding spine image are obtained, and the transformed coordinates are as follows: ; ; ; .

[0049] S335. Based on the transformed coordinates, crop out the pixels within the rectangular area at the corresponding position in the spine image of the book, such as... Figure 7 As shown, the call number image is obtained.

[0050] S336. Following the same method as steps S331 to S335, process the book title rectangular bounding box and the author rectangular bounding box output by the second model respectively to obtain the book title image and the author image.

[0051] Step three yielded the call number image, title image, and author image for each book, all of which are independent of each other.

[0052] Step 4: Use the third model to detect text lines in each of the call number images, output the text line bounding boxes, and capture the text line images.

[0053] Before step four, a preprocessing step of text orientation detection and correction is included. The fifth model is used to determine the text orientation in each call number image. The call number image is input into the fifth model; if the model output is not horizontal, the image is rotated 90 degrees counterclockwise to make the text horizontal. Figure 8 As shown. After this processing, the text direction in all call number images is unified.

[0054] Specifically, the fifth model employs a trained lightweight image classification model, CNSM-Cls (Call Number Symbolic Mapping Classification), based on the PPLCNet architecture. This model is trained as a binary classifier to determine whether the text in the input image is horizontally oriented. The output text orientation classification format is as follows: ,in The confidence levels correspond to 0° horizontal and 90° clockwise directions, respectively. This indicates that a 90° counterclockwise rotation is needed to correct the text orientation, and the image is rotated 90° counterclockwise. The model uses the PPLCNet_x1_0 architecture, scaling several call number images proportionally to a height and width of 224, converting the color channels to BGR, normalizing pixels, and augmenting the images before training to obtain the lightweight image classification model CNSM-Cls.

[0055] Specifically, the third model employs the pre-trained CNSM-Det (Call Number Symbolic Mapping Detection) text detection model based on PPOCRv5_mobile_det. This model effectively detects the position of text lines in an image and outputs the bounding boxes of the text lines. Based on the pre-trained PPOCRv5_mobile_det model, several call number images were proportionally scaled and padded to a height of 224 and a width of 288, their color channels were converted to BGR, and pixel normalization and image augmentation were performed before training. This input size was chosen because the model requires h and w to be integer multiples of 32, reducing inference costs while slightly affecting image quality.

[0056] The orientation-corrected call number image is input into the third model, and the model outputs a rectangular bounding box for each line of text, such as... Figure 9 As shown, the bounding box fits tightly to the top and bottom and left and right boundaries of the characters. Based on the bounding box coordinates, each line of text is cropped sequentially from the call number image. For example... Figure 10 As shown, each text line image contains a single line of characters from the call number. All the captured text line images are sorted in top-to-bottom reading order and sequentially fed into step five for recognition.

[0057] Step 5: Use the fourth model to recognize each of the text lines in the image, and output the initial recognized text containing the mapping symbols, such as... Figure 11 As shown; wherein, the fourth model is trained based on a mapping strategy.

[0058] Specifically, the fourth model employs the pre-trained text recognition model CNSM-Rec (Call Number Symbolic Mapping Recognition) based on PPOCRv5_mobile_rec. Building upon the pre-trained en_PPOCRv5_mobile_rec model, to reduce training costs, the text line image is proportionally scaled to a height of 48 and a width of 320, the color channels are converted to BGR, and pixel normalization and image augmentation are performed before training to obtain the text recognition model CNSM-Rec. During the training phase of the CNSM-Rec text recognition model, a mapping strategy is introduced to effectively avoid the problem of indistinguishability caused by partially missing characters. This partial missingness refers to the loss of a portion of the character's structure due to factors such as label folding.

[0059] The mapping strategy does not require the fourth model to make a hard, either-or judgment when information is insufficient. Instead, it allows the fourth model to classify indistinguishable characters into a fuzzy class and use mapping symbols to represent this uncertainty.

[0060] The specific design process of the mapping strategy is as follows: Character groups are defined. First, a large number of call numbers are analyzed to identify characters that are easily confused or difficult to distinguish when partially missing, and these are grouped into the same character group. For example, when the upper part of the letter B is obscured, the remaining part is very similar to the letters P and R, so B, P, and R are defined as one character group. Similarly, when the lower part of the number 2 is missing, it is difficult to distinguish from the number 3, so 2 and 3 are defined as another character group. Characters within each character group are easily confused with each other.

[0061] Assign a mapping symbol. Assign a unique mapping symbol to each character group. The mapping symbol should be a special symbol that does not appear in the call number to avoid confusion with the actual characters.

[0062] The call number contains a character set of 41 characters: numbers 0-9, uppercase English letters A, Z, and symbols . / -():. Analysis revealed 15 characters that were indistinguishable when partially or completely missing. Mapping pairs were designed as shown in Table 1. If the corresponding original character is not obscured or missing, it is still mapped to the original character. For the other 26 characters in the call number, the partially obscured characters are mapped to the original characters.

[0063] Table 1. Mapping pairs of characters that cannot be distinguished when half-characters or partially missing characters are present. .

[0064] Training data annotation. When constructing the training dataset for the fourth model, for each text line image sample, if a character cannot be uniquely identified due to simulated occlusion or actual absence, it is annotated with the mapping symbol corresponding to its character group. For example, in a call number text line image, the originally clear character sequence is TP312, but the upper part of P is missing due to folding, and the remaining part's shape is between B, P, and R. In this case, P is replaced with the mapping symbol @, i.e., the annotated text is T@312.

[0065] Model Training. The fourth model was trained using the processed training dataset described above. The model no longer learns to forcibly distinguish between B, P, and R in ambiguous situations, but instead outputs the mapping symbol @ when the input image features are so blurred that they cannot be determined among B, P, and R. This significantly reduces the learning difficulty of the model and eliminates the high risk of misclassification caused by forced classification.

[0066] Step 6: For each of the initial recognition texts, based on the correspondence between the mapping symbols and character groups, replace each mapping symbol in the initial recognition text with each character in the character group corresponding to the mapping symbol, thereby generating at least one candidate call number corresponding to the initial recognition text.

[0067] In the initial recognized text obtained in step five, the mapping symbols represent a set of possible characters, rather than a uniquely determined character. Therefore, it is necessary to expand the mapping symbols into specific characters to generate all possible candidate call numbers for subsequent verification.

[0068] For each text line image corresponding to the initial recognition text, iterate through each character. If the current character is a normal character, leave it unchanged; if the current character is a mapping symbol, then according to the correspondence between the mapping symbol and the character group defined in step five, replace each character in its corresponding character group with the mapping symbol in turn, generating a new text sequence. If an initial recognition text contains multiple mapping symbols, then a combination replacement is required, that is, enumerate all possible combinations of characters in all mapping symbols one by one.

[0069] For each of the initially identified texts, the expression for the number of corresponding generated candidate call numbers is: ; in, This represents the total number of mapped symbols in an initial recognized text. Indicates the first The number of characters in the character group corresponding to each mapping symbol Integer and Since call numbers are typically ≤10 characters long, and the number of mapping symbols in each initial recognition text is ≤5, the worst-case number of candidate call numbers generated will be ≤ Secondly, its retrieval efficiency is controllable.

[0070] For example, if the initial recognition text contains two mapping symbols, the first mapping symbol corresponds to a character group with 3 characters, and the second mapping symbol corresponds to a character group with 2 characters, then a total of 3×2=6 candidate call numbers will be generated.

[0071] The candidate results of all text line images in a call number image are merged in order to form at least one complete candidate call number corresponding to that call number.

[0072] Step 7: Compare the at least one candidate call number with the standard call numbers in the standard call number database. If a unique standard call number can be matched, the call number retrieval for the spine is considered successful; otherwise, it is marked as pending manual verification.

[0073] The standard call number database pre-imports all call numbers from the library's collection into the library management system and associates them with corresponding bibliographic information, including title, author, and publisher. The database is indexed to support fast, precise, and fuzzy searches.

[0074] Step seven includes the following specific steps: All candidate call numbers generated in step six are used as query conditions to perform exact matching queries in the standard call number database.

[0075] If the query results match one and only one database record, it means that the current candidate call number is a valid and unique call number. In this case, the retrieval is considered successful, and the book information associated with that record is obtained.

[0076] If none of the candidate call numbers match any database record, it means that the call number on the spine of the book may not be automatically recovered by this method due to severe damage to the label, or the call number itself is not in the standard database. Further attempts should be made to search the database for author and book title information. If a unique call number still cannot be found, the book should be marked as awaiting manual verification, and its shelf location information should be recorded and handed over to library staff for manual processing.

[0077] If any candidate call number matches different database records (i.e., multiple matching standard call numbers exist), it means that the call number information alone cannot uniquely identify the book. In this case, auxiliary matching is initiated, namely: The book title image and author image output by the second model are identified to obtain book title information and author information; The book title and author information are compared with the book information (book title and author) corresponding to the multiple standard call numbers matched. If there is one and only one database record of book information that matches the identified book title and author information, the retrieval is considered successful and the book information is recorded. If there is no matching book information or multiple matching standard call numbers after comparison, it is marked as pending manual verification.

[0078] Step 8, Overall Loop: Repeat until all bookshelves have been processed.

[0079] The following specific experiments illustrate the superiority of the multi-model collaborative book call number identification and correction method of the present invention.

[0080] All models in this experiment were trained and tested based on the following hardware environment: NVIDIA GeForce RTX 3090 graphics processor (24 GB VRAM), dual Intel Xeon Gold 6138 CPUs @ 2.00GHz (80 logical threads in total), 64 GB of memory, and Ubuntu 24.04.3 LTS operating system.

[0081] Based on the differences in the deep learning frameworks used by the models, two software environments were constructed. Environment 1 was used to train the spine detection model BSM-OBB and the call number detection model BTCNA-Det. Its core software versions were Python 3.12 and CUDA Toolkit 13.0, and PyTorch 2.9.0, torchvision 0.24.0, torchaudio 2.9.0, and ultralytics 8.4.22 were installed. Environment 2 was used to train the call number orientation classification model CNSM-Cls, the text line detection model CNSM-Det, and the semi-character recognition model CNSM-Rec. Its core software versions were Python 3.12 and CUDA Toolkit 13.0, and PaddlePaddle-GPU 3.3.0 and PaddleOCR-3.2.0 were deployed.

[0082] Based on the characteristics of each task, corresponding evaluation indicators are used to quantitatively evaluate the performance of each model, and performance comparison charts are drawn to intuitively show the detection and recognition effects.

[0083] 1. Spine detection model BSM-OBB The mean average precision (mAP) was used as the detection performance index. Specifically, the average value of the intersection over union (IoU) threshold from 0.5 to 0.95 with a step size of 0.05 was calculated and denoted as mAP@[0.5:0.95]. The mAP@[0.5:0.95] obtained by this spine detection model was 0.92, indicating that it has high accuracy in locating and regressing the bounding box of the spine region.

[0084] 2. Call Number Detection Model BTCNA-Det The mAP@[0.5:0.95] was also used for evaluation. In this call number detection task, the model achieved an mAP@[0.5:0.95] of 0.93, demonstrating its high-quality detection and location accuracy of the call number area.

[0085] 3. CNSM-Cls Call Number Orientation Classification Model On the validation set, classification accuracy was used as the evaluation metric, defined as the ratio of correctly predicted samples to the total number of samples. This orientation classification model achieved a validation set accuracy of 0.99, demonstrating near-perfect accuracy in determining the orientation of call number text.

[0086] 4. CNSM-Det Text Line Detection Model On the validation set, the quality of text line segmentation is measured by pixel accuracy, which is the ratio of correctly predicted pixels to the total number of pixels in the image. This model achieves a pixel accuracy of 0.93, demonstrating accurate instance-level segmentation of text line regions.

[0087] 5. Semi-character recognition model CNSM-Rec The string recognition performance was evaluated on the validation set using Exact Match Accuracy, which is the proportion of samples where the predicted string completely matches the real string out of the total number of samples. This half-character recognition model achieved an Exact Match Accuracy of 0.968, reflecting its extremely high overall recognition capability for half-character sequences.

[0088] Through the performance evaluation and comparison of the above systems, the effectiveness and advancement of the models provided in this embodiment in library spine image understanding and automatic call number recognition tasks have been fully verified.

[0089] 6. Comparative Validation of Character Mapping Training To verify the improvement in recognition accuracy brought about by the character mapping training strategy, an ablation experiment was further designed. The experiment adopted an edit-based approach. The accuracy of the edit distance is used as an evaluation metric, and the formula is: Accuracy = 1 - Edit Distance / max(Predicted String Length, True String Length), where edit distance refers to the minimum number of operations required to convert one string into another by inserting, deleting, or replacing single characters. On the CNSM-Rec model, a comparison is made between training with and without character mapping, and the comparison results are as follows. Figure 12 As shown.

[0090] Figure 12 In the analysis of unmasked character images, the edit distance accuracy of the model trained without character mapping is calculated based on the model's recognition results and the standard annotations that do not contain mapped characters. Similarly, the edit distance accuracy of the model trained with character mapping is calculated based on the model's recognition results and the standard annotations that do not contain mapped characters. At this point, the difference in edit distance accuracy between the two training strategies is approximately 0.3%, indicating that both models possess good basic recognition capabilities under normal unmasked conditions.

[0091] In images containing occluded characters, the standard annotations include mapped characters. Since models not trained with character mapping cannot recognize the categories of mapped characters, the mapped character portion in their predictions is considered a mismatch with the mapped characters in the annotations, equivalent to an error. Therefore, the edit distance accuracy of this model is calculated based on its recognition results and the annotations containing mapped characters. In contrast, models trained with character mapping compare their recognition results with the annotations containing mapped characters to calculate their edit distance accuracy. Under these test conditions, the edit distance accuracy of models trained with and without character mapping differs significantly, reaching 42.2%. This significant difference indicates that the original general OCR model without character mapping training cannot effectively extract information from the covered character region, while the model trained with character mapping can accurately identify the corresponding character groups, demonstrating outstanding recognition performance advantages in occluded scenarios.

[0092] 7. Comparison of call number retrieval success rates like Figure 13 As shown, on the test set containing samples with masked characters, the end-to-end retrieval success rate of the call number using the mapping training method of this invention is 93.66%, which is 11.87 percentage points higher than the 81.79% without the mapping training method, verifying the significant improvement effect of the delayed decision strategy on the accuracy of call number retrieval.

Claims

1. A multi-model collaborative method for book call number identification and correction, characterized in that, Includes the following steps: S1. Obtain a bookshelf image containing multiple book spines; S2. Use the first model to detect the spine of the bookshelf image, output the rotation bounding box of each spine, and straighten the image within each rotation bounding box through perspective transformation to obtain the straightened spine image of each spine. S3. Using the second model, locate the call number region, title region, and author information region from each of the upright spine images, and extract the call number image, title image, and author image. S4. Use the third model to detect text lines in each of the call number images, output the text line bounding boxes, and capture the text line images. S5. The fourth model is used to recognize each of the text lines in the image, and the initial recognized text containing the mapping symbol is output. The fourth model is trained based on a mapping strategy, which is as follows: multiple character groups are defined, each character group consists of multiple characters that cannot be distinguished from each other due to partial missing characters; a unique mapping symbol is assigned to each character group; when labeling each sample in the training dataset of the fourth model, for characters that cannot be distinguished due to partial missing characters, the character is labeled with the mapping symbol corresponding to its character group. S6. For each of the initial identification texts, according to the correspondence between the mapping symbols and character groups, the mapping symbols in the initial identification text are replaced one by one with each character in the character group corresponding to the mapping symbol, thereby generating at least one candidate call number corresponding to the initial identification text. S7. Compare the at least one candidate call number with the standard call numbers in the standard call number database. If a unique standard call number can be matched, the call number retrieval for the spine of the book is determined to be successful; otherwise, it is marked as pending manual verification.

2. The multi-model collaborative book call number identification and correction method according to claim 1, characterized in that, Step S2 includes the following steps: S21. Input the bookshelf image into the first model, and output the rotation bounding box parameters for each spine of the bookshelf; the rotation bounding box parameters include the center coordinates and width of the rotation bounding box. ,high and rotation angle ; S22. Calculate the coordinates of the four corner points of each of the rotated bounding box parameters. , , as well as ,in The coordinates of the top right corner point, The coordinates of the top left corner point, The coordinates of the bottom left corner point, The coordinates of the bottom right corner point; S23. Establish the correspondence between the four corner points obtained in step S22 and the four corner points of the target horizontal rectangle, solve for the perspective transformation matrix H, and straighten the image within the rotating bounding box according to the perspective transformation matrix to obtain the straightened spine image of each spine; wherein, the coordinates of the four corner points of the target horizontal rectangle are respectively , , , .

3. The multi-model collaborative book call number identification and correction method according to claim 2, characterized in that, Step S3 includes the following steps: S31. All the upright spine images obtained in step S2 are stitched together row by row to form a stitched image; S32. Use the second model to perform target detection on the stitched image and output rectangular bounding boxes for three types of targets: call number, book title, and author. S33. Based on the rectangular bounding boxes of the three types of targets (call number, book title, and author) output, determine the spine image to which each rectangular bounding box belongs. Calculate the coordinates of each rectangular bounding box in its corresponding spine image. Then, based on the calculated coordinates, extract the call number image, book title image, and author image from the corresponding spine image.

4. The multi-model collaborative book call number identification and correction method according to claim 3, characterized in that, Step S31 specifically includes the following steps: S311, Set the set of spine images for correcting the spine as follows: , Indicates the first A spine image of a book that has been turned into a standard version. Integer and Each spine image for turning the book into a standard edition The original width is The height is ; S312, For each of the said spine-correcting images Fill with a preset color along the width direction; the width after filling is... ;in, Indicates the width of the preset fill color; S313, Set the canvas height to and width are And set the canvas background color; S314. Calculate the scaling ratio Each of the upright spine images is scaled proportionally to a height of [height missing]. Width is = This yields a scaled image of the book spine. S315, using the horizontal axis = Starting from the stitching point, the scaled spine image is sequentially stitched along... Place the axis on the canvas; if the x-coordinate of the current stitching position point appears... satisfy Then, after expanding the width of the stitched image, the currently scaled spine is placed into the expanded portion of the stitched image, and the x-coordinate of the next stitching position point is updated. , If the x-coordinate of the current splicing position point appears... satisfy Then calculate the offset. Move the stitched image from the horizontal axis of the canvas Start by placing the image at the center, output the current canvas, then reinitialize the canvas and the stitched image, and continue stitching with the current scaled spine image as the starting image for the reinitialized stitched image.

5. The multi-model collaborative book call number identification and correction method according to claim 4, characterized in that, Step S33 specifically includes the following steps: S331. Obtain the coordinates of the four vertices of the bounding box of the call number rectangle output by the second model, which are respectively , , , ,in, This represents the coordinates of the top-left corner. This represents the coordinates of the top right corner point. This represents the coordinates of the bottom right corner point. This indicates the coordinates of the lower left corner point, with all coordinate values ​​taking the upper left corner of the S31 output image as the origin; S332. Calculate the x-coordinate of the center point of the rectangular bounding box containing the call number. ; S333, Based on the x-coordinate of the center point Determine the spine image to which the rectangular bounding box of the call number belongs. If the x-coordinate of the center point Image on the spine of the book If the bounding box occupies the horizontal coordinate range in the stitched image, it is determined that the bounding box belongs to the spine image. ; S334. Subtract the spine image of the book from the x-coordinate of the bounding box of the call number rectangle. The x-coordinate of the corresponding splicing position point With offset and according to the scaling ratio Convert back to the corresponding coordinates of the spine image of the book to obtain the converted coordinates; S335. Based on the transformed coordinates, extract the call number image from the corresponding spine image. S336. Following the same method as steps S331 to S335, process the book title rectangular bounding box and the author rectangular bounding box output by the second model respectively to obtain the book title image and the author image.

6. A multi-model collaborative book call number identification and correction method according to claim 1 or 5, characterized in that, Before step S4, the text direction of each call number image is detected and corrected. Specifically, the fifth model is used to determine whether the text direction of the call number image is horizontal. If it is not horizontal, the call number image is rotated to the horizontal direction.

7. The multi-model collaborative book call number identification and correction method according to claim 5, characterized in that, In step S6, the expression for the number of candidate call numbers generated for each of the initially identified texts is: ； in, This represents the total number of mapped symbols in an initial recognized text. Indicates the first The number of characters in the character group corresponding to each mapping symbol It is an integer, and .

8. The multi-model collaborative book call number identification and correction method according to claim 6, characterized in that, The specific process of step S7 includes: The at least one candidate call number is compared with the standard call numbers in the standard call number database; If a unique matching standard call number exists, the retrieval is considered successful and the book information is recorded. If no matching standard call number is found, it will be marked as pending manual verification. If multiple matching standard call numbers exist, the title image and author image output by the second model are identified to obtain title information and author information; the title information and author information are then matched with the book information corresponding to the multiple matching standard call numbers; if a unique matching book information is found after the auxiliary matching, the retrieval is considered successful and the book information is recorded; if no matching book information is found after the auxiliary matching or multiple matching standard call numbers still exist, the book information is marked as awaiting manual verification.

9. The multi-model collaborative book call number identification and correction method according to claim 6, characterized in that, The first model is a rotated bounding box detection model based on the YOLO11 architecture; the second model is an object detection model based on the YOLO11 architecture; the third model is a text detection model based on PPOCRv5_mobile_det; the fourth model is a text recognition model based on PPOCRv5_mobile_rec; and the fifth model is an image classification model based on the PPLCNet architecture.

Images