A book image processing method and device, computer equipment and computer program product

Abstract

The application belongs to the technical field of computers, and particularly relates to a book image processing method and device, computer equipment and a computer program product. The application firstly processes multiple original images collected by a visual inventory robot, then extracts image features from a book spine area image set and performs feature matching to obtain matching feature point data between images in the book spine area image set, and finally based on the matching feature point data, adopts a panorama fusion strategy to stitch the images in the book spine area image set. When the panorama fusion stitching result is abnormal, the dynamic linear offset synthesis strategy is automatically switched to for secondary stitching, and a stitched panorama image is output, which can solve the problems of image content overlap, different viewing angles, stitching redundancy, misjudgment, misplacement and poor stability caused by multi-point collection of the visual inventory robot in a library.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a method, apparatus, computer equipment, and computer program product for processing book images. Background Technology

[0002] With the rapid development of smart library construction, the efficiency and accuracy of traditional manual inventory management can no longer meet business needs. The process is time-consuming, labor-intensive, and susceptible to external interference, often requiring library closure and severely disrupting normal circulation. Therefore, automated inventory technology is becoming a trend explored by the library industry. Among these, robot-based automated inventory technology, with its convenience, efficiency, accuracy, and uninterrupted operation, is gradually becoming the preferred inventory solution for libraries, leading library book management into a new stage of real-time and intelligent management.

[0003] The visual inventory robot is an intelligent book inventory robot equipped with multiple high-resolution cameras. It automatically cruises along the bookshelf track in the library and collects image data of the bookshelf shelves at fixed points. The robot's client transmits the collected image data to the visual inventory service system via the network. First, the image undergoes preprocessing and deduplication and fusion calculation. Then, computer vision technology is used to automatically identify the spine content of the books in the images. Finally, the system obtains the inventory location data corresponding to the physical books and the book information through a matching model, and prompts business issues such as misplaced books or books that are not on the shelf.

[0004] Due to the physical space constraints of library bookshelves, the visual inventory robot's patrol process inevitably requires multiple fixed-point data collections per column to ensure the integrity of the image data collected for each column. Therefore, multiple point collections in the same column may lead to overlapping data content between images, resulting in the visual technology recognizing too many book spines and incorrect position sorting. Summary of the Invention

[0005] To address the technical problem that existing visual inventory robots, when performing book inventory tasks in libraries, suffer from overlapping data in the images they collect, resulting in excessive number of book spines being identified and incorrectly sorted, this application provides a book image processing method, apparatus, computer equipment, and computer program product.

[0006] In a first aspect, embodiments of this application provide a book image processing method, the method being applied to a computer device, the method comprising:

[0007] The multiple frames of original images collected by the visual inventory robot are processed to obtain a set of book spine region images that have been normalized in size; wherein, the multiple frames of original images are image data collected by the visual inventory robot during the execution of the book inventory task. Image features are extracted from the set of images of the spine region of the books and feature matching is performed to obtain matching feature point data between images in the set of images of the spine region of the books. Based on the matching feature point data, a panoramic fusion strategy is used to stitch together the images in the set of images of the spine region of the books. When the panoramic fusion stitching result is abnormal, it automatically switches to a dynamic linear offset synthesis strategy for secondary stitching and outputs the stitched panoramic image.

[0008] The technical advantages of this application are as follows: First, it processes multiple frames of original images collected by the visual inventory robot. Then, it extracts image features from the set of images of the book spine regions and performs feature matching to obtain matching feature point data between images in the set of images of the book spine regions. Finally, based on the matching feature point data, it uses a panoramic fusion strategy to stitch the images in the set of images of the book spine regions. When the panoramic fusion stitching result is abnormal, it automatically switches to a dynamic linear offset synthesis strategy for secondary stitching, outputting the stitched panoramic image. This constructs a complete automated process from original image processing to accurate stitching output, ensuring the seamlessness, completeness, and high accuracy of the final bookshelf "panoramic image," providing a reliable data foundation for subsequent book recognition. This can solve the problems of overlapping image content and different perspectives caused by multi-point acquisition in library visual inventory robots, leading to stitching redundancy, misjudgment, misalignment, and poor stability.

[0009] In some embodiments, processing the multiple frames of raw images acquired by the visual inventory robot to obtain a set of size-normalized book spine region images includes: The original images of the multiple frames are subjected to noise reduction processing to obtain a preprocessed image; Extract sub-images containing only the spines of books from the preprocessed image; Each sub-image is normalized to obtain a set of book spine region images that have been size normalized.

[0010] The technical effect of this embodiment is that by obtaining a set of book spine region images that have been normalized in size, this embodiment provides standardized, clean, and effective input data for subsequent feature calculation.

[0011] In some embodiments, the noise reduction processing of the multiple original images to obtain a preprocessed image includes: The original images of the multiple frames are sequentially subjected to geometric transformation, sharpening and enhancement, grayscale conversion and filtering and denoising to obtain a preprocessed image; wherein, the geometric transformation includes adjusting the spine direction in the original image to the vertical direction and scaling it proportionally based on a preset image size; The step of extracting a sub-image containing only the spine of the book from the preprocessed image includes... The pre-trained spine target detection model is used to identify the pre-processed image to obtain the coordinate set of the spine region of the book. The coordinate set of the spine region of the book is sorted and its integrity is checked to obtain a sorted coordinate set. Then, a sub-image containing only the spine of the book is extracted from the sorted coordinate set using image segmentation technology. The normalization process for each sub-image to obtain a set of book spine region images with normalized size includes: The smallest sub-image is determined from all the sub-images. Then, the size of each sub-image is compared with the smallest sub-image, and each sub-image is uniformly scaled to the same size as the smallest sub-image to form a set of book spine region images that have been normalized in size.

[0012] The technical advantages of this embodiment are as follows: through a standardized preprocessing process, the quality and consistency of the input data are significantly improved, a large number of environmental interference factors are eliminated, and a solid foundation is laid for high-precision feature matching; it solves problems such as uneven lighting, noise, inconsistent spine direction, size differences and interference from invalid background areas in the original image, and provides standardized, clean and effective input data for subsequent feature calculation.

[0013] In some embodiments, the step of extracting image features from the set of images of the book spine region and performing feature matching to obtain matching feature point data between images in the set of images of the book spine region includes: The feature extraction network is used to process each frame of the image set in the specific region to extract the feature points and corresponding feature descriptors of each frame; Using a feature matching network, common feature descriptors between adjacent frames in the book spine region image set are matched, and successfully matched feature point pairs are output to form matching feature point data between images in the book spine region image set.

[0014] The technical advantages of this embodiment are as follows: by using a feature extraction network and a feature matching network, the accuracy and stability of feature extraction and matching in complex scenes are significantly improved. It can accurately identify overlapping areas between images and effectively reduce false matching. It can solve the problem of low matching accuracy and poor robustness of traditional feature algorithms (such as SIFT) in book spine images with similar textures and complex lighting.

[0015] In some embodiments, the panoramic fusion strategy includes: Based on the matching feature point data, the projection transformation relationship between images in the set of images of the book spine region is obtained by calculating the homography matrix; Based on the projection transformation relationship, perspective transformation is performed on the images in the set of images of the spine region of the book, and the size of the new canvas is determined according to the coordinates of the perspective-transformed images. Based on the size of the new canvas and the projection transformation relationship, each of the perspective-transformed images is mapped onto the new canvas to achieve coordinate space alignment, thereby obtaining a spatially aligned intermediate image. Calculate the overlap region mask between the spatially aligned intermediate images, and use the overlap region mask to fuse the spatially aligned intermediate images to obtain and output the first stitched image.

[0016] The technical advantages of this embodiment are as follows: Under ideal conditions, this embodiment can generate a complete, aesthetically pleasing, and geometrically accurate panoramic image of a bookshelf, providing high-quality input for visual recognition. Based on accurate feature matching, it can achieve seamless and smooth stitching of multi-view images, generating a visually coherent and distortion-free panoramic image.

[0017] In some embodiments, when a canvas anomaly or image distortion is detected in the first stitched image, the dynamic linear offset synthesis strategy is triggered, including: Based on the matching feature point data, the projection transformation relationship between images in the set of images of the book spine region is obtained by calculating the homography matrix; Based on the projection transformation relationship, perspective transformation is performed on the images in the set of images of the spine region of the book, and the size of the new canvas is determined according to the coordinates of the perspective-transformed images. Based on the size of the new canvas and the projection transformation relationship, each of the perspective-transformed images is mapped onto the new canvas to achieve coordinate space alignment, resulting in a spatially aligned intermediate image.

[0018] The technical advantage of this embodiment is that the computer system first executes a panoramic fusion stitching strategy. If the generated first stitched image is of acceptable quality (e.g., without severe distortion or misalignment), it is used as the final output. If the system detects an anomaly in the stitching result (e.g., abnormal canvas size or obvious misalignment of image content), it automatically switches to a dynamic linear offset synthesis stitching strategy, executes a compensation stitching process, and uses the output second stitched image as the final panoramic stitched image. This hybrid strategy ensures the robustness and reliability of the entire image stitching and synthesis process.

[0019] Secondly, this application also proposes a book image processing apparatus, which is applied to a computer device, and the apparatus includes: The image data processing module is used to process multiple frames of raw images collected by the visual inventory robot to obtain a set of book spine area images that have been normalized in size. The image feature calculation module is used to extract image features from the set of images of the spine region of the book and perform feature matching to obtain matching feature point data between images in the set of images of the spine region of the book. The image stitching and synthesis module is used to stitch the images in the set of images of the spine region of the book based on the matching feature point data and a panoramic fusion strategy. When the panoramic fusion stitching result is abnormal, it automatically switches to the dynamic linear offset synthesis strategy for secondary stitching and outputs the stitched panoramic image.

[0020] Thirdly, this application also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer device is signal-connected to a visual inventory robot. The processor of the computer device acquires multiple frames of original images collected by the visual inventory robot, and the processor executes the computer program to implement the method described in the first aspect above.

[0021] In some embodiments, the computer device is a remote server, or the computer device is deployed inside the visual inventory robot.

[0022] Fourthly, this application also proposes a computer program product having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0023] The technical effects of the second to fourth aspects mentioned above are similar to those of the first aspect and will not be elaborated upon here. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a simplified structural diagram of the computer device provided in the embodiments of this application; Figure 2 This is a simplified structural diagram of the visual inventory robot provided in the embodiments of this application; Figure 3 This is a schematic flowchart of a book image processing method provided in an embodiment of this application; Figure 4This is a simplified structural diagram of a book image processing device provided in an embodiment of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0030] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] like Figure 1 As shown, this application embodiment provides a computer device, including a memory 1006, a processor 1001, and a computer program stored in the memory and executable on the processor 1001. The computer device is signal-connected to a visual inventory robot. The processor 1001 of the computer device acquires multiple frames of raw images captured by the camera 1004 of the visual inventory robot. When the processor 1001 executes the computer program, it implements the steps of a book image processing method described in the following embodiments.

[0033] In some embodiments, Figure 1 The computer device shown can be a remote server, connected to the visual inventory robot via remote network communication. In other embodiments, the computer device can be directly deployed inside the visual inventory robot, such as... Figure 2 As shown, the visual inventory robot can directly interact with computer devices through the internal communication bus 1002.

[0034] For example, camera 1004 can be a high-definition camera. During the process of the library visual inventory robot performing the book inventory task, camera 1004 can be used to take pictures of each batch of books to be inventoried on the bookshelves in the library to collect bookshelf layer images.

[0035] The processor 1001 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0036] In view of the above-mentioned defects mentioned in the background technology, refer to Figure 3 This application provides a book image processing method, which is... Figure 1 The method, executed by the computer device shown, mainly includes steps S1 to S3: Step S1 (Image Data Processing): Process the multiple frames of raw images collected by the visual inventory robot to obtain a set of size-normalized images of the book spine region. In specific applications, the first stage of this application embodiment preprocesses multiple frames of original images collected by the visual inventory robot, the second stage extracts the book spine region from the collected multiple frames of original images, and the third stage performs size normalization on each collected image to obtain a set of specific region images with uniform size, so as to avoid feature differences caused by size issues.

[0037] Step S2 (Image Feature Calculation): Extract image features from the set of images of the book spine region and perform feature matching to obtain matching feature point data between images in the set of images of the book spine region; In this embodiment, since traditional feature algorithms cannot accurately extract image feature data with complex lighting and multiple perspectives, resulting in significant feature matching errors and instability, this embodiment can employ an end-to-end architecture image neural network algorithm model processing technology to extract and match features from the image set of the specific region, obtaining matching feature point data between multiple frames of images. Through attention mechanism and optimal transmission matching, it significantly improves the accuracy and robustness of image stitching.

[0038] Step S3 (Image Stitching and Composition): Based on the matching feature point data, the images in the set of images of the spine region of the book are stitched together using a panoramic fusion strategy. When the panoramic fusion stitching result is abnormal, the system automatically switches to a dynamic linear offset synthesis strategy for secondary stitching and outputs the stitched panoramic image.

[0039] The technology includes stitching strategies such as panoramic fusion and dynamic linear offset synthesis. Panoramic fusion stitching algorithm is preferred to ensure the seamlessness and integrity of the image stitching result. Only when there is an anomaly in the panoramic fusion stitching result will the algorithm automatically switch to dynamic linear offset synthesis stitching strategy to solve the problem of deduplication data coverage in redundant areas between images and ensure the reliability of stitching data. Therefore, these two stitching strategies are used in combination and integrated into this patented technology solution. The panoramic fusion strategy refers to a stitching method that performs perspective transformation and fusion based on the geometric projection relationship between images; while the dynamic linear offset synthesis strategy refers to a stitching method that calculates the image overlap width and performs horizontal offset coverage according to this width; finally, a stitched panoramic image is generated. The method described in this application can solve the problems of image content overlap and inconsistent perspectives caused by multi-point acquisition in library visual inventory robots, leading to stitching redundancy, misjudgment, misalignment, and poor stability. By closely linking the three steps of image data processing in step S1, image feature calculation in step S2, and image stitching synthesis in step S3, a complete automated process from raw image processing to accurate stitching output is constructed, ensuring the seamlessness, completeness, and high accuracy of the final bookshelf "panoramic image," providing a reliable data foundation for subsequent book recognition.

[0040] In one embodiment, step S1 may further include steps S11 to S13: Step S11: Perform noise reduction processing on the multiple frames of original images to obtain a preprocessed image; In some embodiments, the multiple original images can be subjected to geometric transformation, sharpening enhancement, grayscale conversion, and filtering and denoising processing in sequence to obtain a preprocessed image; wherein, the geometric transformation includes uniformly adjusting the spine direction in the image to the vertical direction, and scaling proportionally based on a preset image size (for example, a minimum image size can be set); As an example, in step S11a of the "geometric transformation" process, the images acquired by the visual inventory robot can be subjected to geometric transformations such as rotation and scaling. Typically, the vertical orientation of the book spines is taken as the consistent direction. For books with inconsistent spine orientations, the corresponding images need to be rotated and calculated to make them consistent. Next, it is calculated whether the sizes in the images are the same. Using the smallest image size as the baseline, other images of different sizes are proportionally transformed, and then the corresponding image sizes are calculated by scaling. Finally, image data with consistent angles and sizes is obtained.

[0041] Furthermore, as an example, for the relevant step S11b of "sharpening enhancement", the image data processed in the "geometric transformation" step S11a is subjected to sharpening enhancement calculation processing; for example, sharpening enhancement can be achieved by customizing the kernel parameters of the sharpening edge operator, and a new image is obtained after image sharpening convolution calculation processing; making the edge texture and feature details of the spine in its image area clearer and more obvious.

[0042] Furthermore, as an example, for the relevant step S11c of "grayscale conversion", the image data processed in the "sharpening and enhancement" step S11b is uniformly converted to grayscale to eliminate the interference of the RGB color space in the image on the image detection process; in the specific implementation, the following formula (1) is used for calculation: Gray(i, j) = α R(i, j) + β G(i, j) + γ B(i, j)(Formula 1) Where (i, j) are pixel coordinates, Gray(i, j) is the output gray value, R(i, j), G(i, j), and B(i, j) are the red, green, and blue component values ​​of the corresponding coordinates of the input color image, and α, β, and γ are weight coefficients, satisfying α + β + γ = 1; the weight coefficients α, β, and γ are adaptively adjusted according to the image region features.

[0043] Furthermore, as an example, for the relevant step S11d in "filtering and denoising", the image data processed in the "grayscale conversion" step S11c is subjected to filtering and denoising calculation. For example, the filtering and denoising algorithm in this embodiment can be Gaussian filtering algorithm, which performs weighted averaging based on the weight kernel generated by the Gaussian function, which can more gently eliminate noise points in the image, making its blurring effect and image texture features more natural.

[0044] It is understood that step S11 in this application embodiment is to perform a series of preprocessing on the source image to reduce noise and image irregularity interference factors, and obtain a preprocessed image to ensure the stability of the detection and extraction results of specific regions in the second stage.

[0045] Step S12: Extract sub-images containing only the spines of books from the preprocessed image; In one embodiment, a pre-trained spine target detection model can be used to identify the preprocessed image to obtain a set of coordinates for the spine region of the book; the set of coordinates for the spine region of the book is sorted and its integrity is verified to obtain a sorted set of coordinates; and a sub-image containing only the spine of the book is extracted from the sorted set of coordinates using image segmentation technology. In this embodiment of the application, as a preferred implementation, the pre-trained spine target detection model can be implemented based on the YOLO algorithm. The YOLO algorithm of the system is used to detect the target content of the spine region of the book in the pre-processed image and segment the region sub-image to reduce the interference of invalid blank areas on feature matching.

[0046] As an example, the preprocessed image in step S11 can be processed by the YOLO model for book spine detection; by loading the pre-trained book spine target detection model file of the system, inputting the preprocessed image, and through model inference calculation, a set of coordinates corresponding to a series of target book spines in the preprocessed image can be obtained; Because the coordinate set corresponding to the above series of target book spines has problems such as disordered sequence numbers and missing coordinates, this application embodiment performs coordinate transformation on the coordinate set corresponding to the above series of target book spines and re-sorts the coordinates. The sorted coordinate set is compared with the image width to see if there are any missing coordinate points. If there are any missing coordinate points, the coordinate points in the width direction are supplemented. The result processed in step b is subjected to coordinate region segmentation; by inputting the result image of the source data preprocessing in step 1) and the result coordinate point set in step b, the coordinate corresponding region image is calculated using polygon mask vision technology.

[0047] Step S13: Normalize each sub-image to obtain a set of book spine region images that have been size normalized; As an example, the smallest sub-image can be determined from all the sub-images, and then the size of each sub-image can be compared with the smallest sub-image. Each sub-image can then be uniformly scaled to the same size as the smallest sub-image to form a set of book spine region images that have been normalized in size. It is understandable that the third stage of step S13 is to verify whether the sub-images after segmentation are consistent in size. For inconsistent sizes, normalization processing is performed to achieve unified conversion, so as to avoid feature differences caused by size issues.

[0048] In one embodiment, step S2 may further include steps S21 and S22: Step S21: Use a feature extraction network to process each frame of the image set in the specific region, and extract the feature points and corresponding feature descriptors of each frame; It is understood that step S21 uses a fully convolutional neural network algorithm model for feature extraction, such as SuperPoint, a deep learning-based self-supervised feature point detection and descriptor extraction algorithm. The feature extraction network related to SuperPoint technology processes each frame of the image set for the specific region. In its specific implementation, this mainly includes the following sub-steps: Sub-step S21a: Define the parameter values ​​of the SuperPoint feature extraction network, including the non-maximum suppression radius (nms_radius), keypoint confidence threshold (keypoint_threshold), and maximum number of keypoints to retain (max_keypoints). Sub-step S21b: Substitute the parameter values ​​from sub-step S21a into the method function of the SuperPoint feature extraction network, and automatically identify the GPU or CPU call according to the hardware device of the computer device executing the book image processing method of this application to complete the object initialization.

[0049] Sub-step S21c: Input new image data for a specific region, convert it into SuperPoint feature recognition parameter format, pass the parameters to the object initialized in sub-step S21b, call the method and complete the inference recognition; Sub-step S21d: Obtain the result of sub-step S21c, parse and convert it into the format of the dataset to be matched, which includes image data, keypoints, descriptors, and scores of each frame. Step S22: Using a feature matching network, the common feature descriptors between adjacent frames in the book spine region image set are matched, and the successfully matched feature point pairs are output to form the matching feature point data between images in the book spine region image set.

[0050] It is understandable that this step uses a feature matching network model based on graph neural networks for feature matching, such as SuperGlue technology. In its specific implementation, it can mainly include the following sub-steps: Sub-step S22a defines and assigns SuperGlue model parameter values, including the number of Sinkhorn algorithm iterations (sinkhorn_iterations) and the matching confidence threshold (match_threshold). Sub-step S22b: Select and assign values ​​to the pre-trained weight model parameters, for example, parameter assignment: weights=outdoor; Sub-step S22c takes the parameter results from sub-steps S22a and S22b and substitutes them into the method function of the SuperGlue feature matching network. Then, it automatically identifies whether the hardware device is GPU or CPU and calls the function to complete the object initialization. Sub-step S22d converts the image data, keypoints, descriptors, and scores of each frame obtained in step S21 into SuperGlue matching parameter format, passes the parameters to the object that initializes the results in sub-step S22c, calls the method, and completes feature matching. Sub-step S22e involves acquiring the result of step S22d, parsing the returned dataset of feature matching, and extracting the data of the same matching keypoints. The extracted data of the same matching keypoints is used as the matching feature point data between images in the set of images of the spine region of the book.

[0051] The technical effect of this embodiment is that traditional feature algorithms cannot accurately extract image feature data with complex lighting and multiple perspectives, resulting in significant feature matching errors and instability. By combining feature extraction and similar region feature matching algorithms, especially by using an end-to-end architecture image neural network algorithm model (SuperGlue) processing technology, the accuracy and robustness of image stitching are significantly improved through attention mechanisms and optimal transmission matching.

[0052] In one embodiment, the panoramic fusion strategy for step S3 mentioned above may include the following sub-steps: Step S31A (Feature Homography Matrix Transformation): Based on the matching feature point data, the projection transformation relationship between images in the set of images of the book spine region is obtained by calculating the homography matrix; In the specific implementation, a new image of the spine region of the book and feature point data matching the same features are used as basic parameters. A set of feature matching data is calculated through homography matrix transformation (findHomography) to generate a projection transformation matrix object that maps the source point to the target point. Step S31B (Calculation of new canvas size): Based on the projection transformation relationship, perform perspective transformation on the images in the set of images of the spine region of the book, and determine the size of the new canvas according to the coordinates of the transformed images; In the specific implementation, the result object of the projection transformation relationship calculated in step S31A is processed by point set perspective transformation calculation (perspectiveTransform) to obtain the coordinate corner point object corresponding to the feature data, and the new image canvas size is calculated. Step S31C (Spatial Coordinate Transformation): Based on the size of the new canvas and the projection transformation relationship, each of the perspective-transformed images is mapped onto the new canvas to achieve coordinate space alignment, resulting in a spatially aligned intermediate image.

[0053] In the specific implementation, the results calculated in steps S31A and S31B are processed by feature point translation transformation matrix. Based on the translation matrix object and the new image parameter data of a specific region, the image data is calculated through perspective transformation (warpPerspective) to achieve spatial coordinate transformation and obtain a spatially aligned intermediate image. Step S31D (Overlapping region mask calculation and fusion region output of new image): Calculate the overlapping region mask between the spatially aligned intermediate images, and use the overlapping region mask to fuse the spatially aligned intermediate images to obtain and output the first stitched image.

[0054] In the specific implementation, the overlapping area mask between the spatially aligned intermediate images is calculated, the overlapping area mask is used as a parameter, and then the non-overlapping areas of each image are written into the same canvas to achieve seamless image fusion. Finally, a new panoramic image with successful stitching is output. In one embodiment, in step S3 above, when a canvas abnormality or image distortion is detected in the first stitched image, the dynamic linear offset synthesis strategy is triggered. It is understood that in step S3, the dynamic linear offset synthesis strategy is a parallel alternative to the panoramic fusion strategy, rather than a subsequent step. It is a processing path restarted after the panoramic fusion strategy is determined to be abnormal, using the dynamic linear offset synthesis strategy for compensation and stitching to resolve data redundancy and duplication issues.

[0055] In this embodiment, the dynamic linear offset synthesis strategy may include the following sub-steps: Step S32a (Dynamic Binary Segmentation): If the matching feature point data does not meet the preset requirements, the binary segmentation method is used to iteratively segment each image in the set of images of the book spine region, and the step of extracting image features from the set of images of the book spine region and performing feature matching is repeated on the segmented sub-images until new effective matching feature point data between the images in the set of images of the book spine region is obtained; the new effective matching feature point data is feature point data that meets the preset requirements; In a specific implementation, the image set of the specific region and the corresponding matching feature point data in the aforementioned steps S1 and S2 are obtained. If the matching feature point data does not meet the preset requirements (for example, if the data with the same feature matching is empty), the book spine region image can be automatically segmented using a binary search method. After segmentation, adjacent sub-images are taken for image feature extraction and matching processes to obtain the final data with the same feature matching. This process is repeated (i.e., step S2 in the aforementioned embodiment is re-executed, or steps S21 and S22 in the aforementioned embodiment are re-executed), with a maximum of 3 binary search processes; until new valid matching feature point data between images in the book spine region image set is obtained (for example, until the data with the same feature matching is not empty).

[0056] Step S32b (Size of overlapping region for same feature conversion): Calculate the width of the overlapping region between adjacent images in the book spine region image set based on the new effective matching feature point data; In the specific implementation, this step can use the new effective matching feature point data from step S32a as the basic parameters, calculate the same feature values ​​through homography matrix transformation (findHomography) to generate homography moment objects, and then perform perspective transformation calculation on the homography moment objects to obtain the coordinate corner points of the projection area. The minimum and maximum values ​​of the coordinate corner points of the projection area are extracted and converted into the corresponding size result (i.e., the width of the overlapping area).

[0057] Step S32c (Original image shifts left and right to cover composite image): Construct a new canvas, and sequentially shift and cover each image in the set of images of the spine region of the book in the horizontal direction according to the width of the overlapping area to complete the stitching, and obtain and output the second stitched image.

[0058] In the specific implementation, the size result calculated in step S32b is used as the width of the overlapping area to construct a new canvas. The first new image of a specific area is filled first, and then the canvas content is covered by shifting to the left or right according to the overlap size. In this way, the stitching process of all images is completed, and the second stitched image is generated and output.

[0059] The technical advantage of this embodiment is that the computer system first executes a panoramic fusion stitching strategy. If the generated first stitched image is of acceptable quality (e.g., without severe distortion or misalignment), it is used as the final output. If the system detects an anomaly in the stitching result (e.g., abnormal canvas size or obvious misalignment of image content), it automatically switches to a dynamic linear offset synthesis stitching strategy, executes a compensation stitching process, and uses the output second stitched image as the final panoramic stitched image. This hybrid strategy ensures the robustness and reliability of the entire image stitching and synthesis process.

[0060] On the other hand, such as Figure 4 As shown, this application also proposes a book image processing apparatus, the apparatus being deployed in... Figure 1 or Figure 3 The computer equipment involved in the process, the device includes: Image data processing module 01 is used to process multiple frames of raw images collected by the visual inventory robot to obtain a set of book spine area images that have been normalized in size. Image feature calculation module 02 is used to extract image features from the set of images of the spine region of the book and perform feature matching to obtain matching feature point data between images in the set of images of the spine region of the book. The image stitching and synthesis module 03 is used to stitch the images in the set of images of the spine region of the book based on the matching feature point data and using a panoramic fusion strategy. When the panoramic fusion stitching result is abnormal, it automatically switches to the dynamic linear offset synthesis strategy for secondary stitching and outputs the stitched panoramic image.

[0061] It should be noted that the book image processing device in this embodiment can represent a chip, which can be deployed in... Figure 1 or Figure 3 The computer equipment involved.

[0062] It should be noted that the information interaction and execution process between the above-mentioned devices / modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0064] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0065] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0066] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0067] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for processing book images, characterized in that, The method is applied to a computer device, and the method includes: The multiple frames of original images collected by the visual inventory robot are processed to obtain a set of book spine region images that have been normalized in size; wherein, the multiple frames of original images are image data collected by the visual inventory robot during the execution of the book inventory task. Image features are extracted from the set of images of the spine region of the books and feature matching is performed to obtain matching feature point data between images in the set of images of the spine region of the books. Based on the matching feature point data, a panoramic fusion strategy is used to stitch together the images in the set of images of the spine region of the books. When the panoramic fusion stitching result is abnormal, it automatically switches to a dynamic linear offset synthesis strategy for secondary stitching and outputs the stitched panoramic image.

2. The method according to claim 1, characterized in that, The process of processing multiple frames of raw images collected by the visual inventory robot to obtain a set of book spine region images that have been normalized in size includes: The original images of the multiple frames are subjected to noise reduction processing to obtain a preprocessed image; Extract sub-images containing only the spines of books from the preprocessed image; Each sub-image is normalized to obtain a set of book spine region images that have been size normalized.

3. The method according to claim 2, characterized in that, The step of performing noise reduction processing on the multiple original images to obtain a preprocessed image includes: The original images of the multiple frames are sequentially subjected to geometric transformation, sharpening and enhancement, grayscale conversion and filtering and denoising to obtain a preprocessed image; wherein, the geometric transformation includes adjusting the spine direction in the original image to the vertical direction and scaling it proportionally based on a preset image size; The step of extracting a sub-image containing only the spine of the book from the preprocessed image includes... The pre-trained spine target detection model is used to identify the pre-processed image to obtain the coordinate set of the spine region of the book. The coordinate set of the spine region of the book is sorted and its integrity is checked to obtain a sorted coordinate set. Then, a sub-image containing only the spine of the book is extracted from the sorted coordinate set using image segmentation technology. The normalization process for each sub-image to obtain a set of book spine region images with normalized size includes: The smallest sub-image is determined from all the sub-images. Then, the size of each sub-image is compared with the smallest sub-image, and each sub-image is uniformly scaled to the same size as the smallest sub-image to form a set of book spine region images that have been normalized in size.

4. The method according to claim 1, characterized in that, The step of extracting image features from the set of images of the book spine region and performing feature matching to obtain matching feature point data between images in the set of images of the book spine region includes: The feature extraction network is used to process each frame of the image set in the specific region to extract the feature points and corresponding feature descriptors of each frame; Using a feature matching network, common feature descriptors between adjacent frames in the book spine region image set are matched, and successfully matched feature point pairs are output to form matching feature point data between images in the book spine region image set.

5. The method according to any one of claims 1 to 4, characterized in that, The panoramic fusion strategy includes: Based on the matching feature point data, the projection transformation relationship between images in the set of images of the book spine region is obtained by calculating the homography matrix; Based on the projection transformation relationship, perspective transformation is performed on the images in the set of images of the spine region of the book, and the size of the new canvas is determined according to the coordinates of the perspective-transformed images. Based on the size of the new canvas and the projection transformation relationship, each of the perspective-transformed images is mapped onto the new canvas to achieve coordinate space alignment, thereby obtaining a spatially aligned intermediate image. Calculate the overlap region mask between the spatially aligned intermediate images, and use the overlap region mask to fuse the spatially aligned intermediate images to obtain and output the first stitched image.

6. The method according to claim 5, characterized in that, If a canvas anomaly or image distortion is detected in the first stitched image, the dynamic linear offset synthesis strategy is triggered, including: Based on the matching feature point data, the projection transformation relationship between images in the set of images of the book spine region is obtained by calculating the homography matrix; Based on the projection transformation relationship, perspective transformation is performed on the images in the set of images of the spine region of the book, and the size of the new canvas is determined according to the coordinates of the perspective-transformed images. Based on the size of the new canvas and the projection transformation relationship, each of the perspective-transformed images is mapped onto the new canvas to achieve coordinate space alignment, resulting in a spatially aligned intermediate image.

7. A book image processing device, characterized in that, The method is applied to a computer device, the apparatus comprising: The image data processing module is used to process multiple frames of raw images collected by the visual inventory robot to obtain a set of book spine area images that have been normalized in size. The image feature calculation module is used to extract image features from the set of images of the spine region of the book and perform feature matching to obtain matching feature point data between images in the set of images of the spine region of the book. The image stitching and synthesis module is used to stitch the images in the set of images of the spine region of the book based on the matching feature point data and a panoramic fusion strategy. When the panoramic fusion stitching result is abnormal, it automatically switches to the dynamic linear offset synthesis strategy for secondary stitching and outputs the stitched panoramic image.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer device is signal-connected to the visual inventory robot. The processor of the computer device acquires multiple frames of original images collected by the visual inventory robot, and the processor executes the computer program to implement the method as described in any one of claims 1 to 6.

9. The computer device as claimed in claim 8, characterized in that, The computer device is a remote server, or the computer device is deployed inside the visual inventory robot.

10. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images