Book correction and segmentation method suitable for intelligent library

Abstract

The invention provides a book correction and segmentation method suitable for an intelligent library, and the method comprises the steps: detecting a book region in a distorted image through a deep Hough transform network, extracting a semantic line, and obtaining a quadrangle corner coordinate; correcting the image by using a perspective transformation technology according to the angular point coordinates; and finally, carrying out semantic segmentation on the corrected image by adopting an improved UNet network. VGG16 is introduced into the improved UNet network to serve as an encoder to enhance the feature expression ability, Haar wavelet down-sampling is combined to improve feature representation, a space and channel dual attention mechanism (CBAM) is used for optimizing feature extraction, and high-precision image reconstruction is achieved through dynamic up-sampling (DySample), so that the effect of precise segmentation is achieved. According to the method, image distortion caused by the change of the shooting angle can be effectively corrected, excellent performance is shown in a spine segmentation task, and reliable technical support is provided for automatic inventory of an intelligent library.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a book correction and segmentation method suitable for an intelligent library. Background Art

[0002] With the rapid development of digital libraries, computer vision-based automated book management has gradually become an important means to improve library service efficiency. However, due to the diversity of book arrangements and the perspective distortion and uneven lighting that may occur during the shooting process, traditional image processing methods are difficult to achieve ideal results in the correction and segmentation of book images. Therefore, the research on accurate correction and segmentation of book images has become a key challenge in the application of smart libraries.

[0003] Book image correction is a key step in processing distorted images in smart libraries. Since the captured images often have problems such as perspective distortion, tilt or rotation, these distortions will cause the spine lines in the image to deviate from the horizontal or vertical direction, thus affecting subsequent image segmentation and recognition tasks. Traditional book correction methods usually rely on geometric transformation, line detection or localization algorithms of specific spine features. However, in complex scenes (such as multi-row book arrangement or occlusion between books), these methods often do not work well. With the increasing demand for high precision and high efficiency in smart libraries, efficient distortion correction has become a major challenge. In order to solve these problems, geometric feature extraction algorithms based on deep learning have gradually become an important research direction for book correction.

[0004] Book segmentation is an important research field in image processing of smart libraries. With the acceleration of library digitization, the demand for automated management systems continues to increase. Book segmentation can accurately identify and extract the spine area in the image, thereby achieving efficient management of book information, such as automated inventory, book borrowing and returning tracking, and book positioning. This technology not only improves the management efficiency of the library, but also provides users with a more convenient service experience. Book segmentation methods can be mainly divided into methods based on traditional book features and methods based on deep learning.

[0005] Book spine segmentation based on traditional methods: N.Tabassum et al. 0A segmentation method for the spine of multi-cell books is proposed. This method can use morphology to remove small areas of the image to obtain accurate spine edges. However, this method can only detect thicker and well-arranged books and cannot be applied to thin books or books that are placed in a disorderly manner. (For details, see [1] N. Tabassum, S. Chowdhury, MK Hossen and SU Mondal, "An approach to recognize book title from multi-cell bookshelf images," 2017 IEEE International Conference on Imaging, Vision & Pattern Recognition (icIVPR), Dhaka, Bangladesh, 2017, pp. 1-6.)

[0006] Book spine segmentation based on deep learning: Zhang et al. proposed an improved Deeplab v3 plus network, combining the DenseASPP framework and strip pooling layer to enhance the book spine features, and borrowed the multi-head self-attention mechanism in Vision Transformer to improve the long-distance feature capture ability, thereby improving the segmentation effect of dense, large aspect ratio and inclined targets. However, the high computational complexity of DenseASPP and self-attention mechanism leads to poor real-time performance of this method, making it difficult to efficiently process a large number of book images, and the complex model structure also increases the difficulty of training and deployment. (For details, see [2] Ji Xiaofei, Zhang Kexin, Tang Lirong. Improved Deeplab v3 plus network book spine segmentation algorithm [J / OL]. Computer Applications: 1-7 [2023-08-11]. http: / / kns.cnki.net / kcms / detail / 51.1307.TP.20230330.1057.004.html.)

[0007] In general, the existing book image correction algorithms have poor adaptability and are difficult to adaptively process distorted images in complex scenes. At the same time, the existing book spine segmentation algorithms perform well when the books are neatly arranged, but the effect is significantly reduced when processing distorted images taken due to camera angle problems. These limitations make the automated inventory technology in smart libraries still face many challenges and become an important issue that needs to be studied in depth. Summary of the invention

[0008] In view of this, the object of the present invention is to provide a book correction and segmentation method suitable for an intelligent library, so as to solve the problems of distortion correction and spine segmentation in book images taken by monitoring equipment.

[0009] The technical solution of the present invention is: a book correction and segmentation method suitable for intelligent libraries, comprising:

[0010] S1: Use the deep Hough transform network to extract the semantic lines of the book area and obtain the coordinates of the four corner points;

[0011] S2: Through the perspective transformation module, the irregular quadrilateral book area defined by the coordinates of the four corner points is geometrically transformed; the irregular quadrilateral is adjusted to a regular rectangular area, so as to obtain a frontal book image without background interference;

[0012] S3: Apply the improved UNet network to segment the book spine of the book image obtained in S2.

[0013] Specifically, S1 includes:

[0014] S11: Use a convolutional neural network encoder to process the input image and extract pixel-level deep features;

[0015] S12: Convert pixel-level deep features from feature space to parameter space through deep Hough transform;

[0016] S13: Processing using a context-aware line detector, performing a convolution operation in the parameter space, aggregating and analyzing the context features of the candidate lines and their surroundings, and detecting the parameters;

[0017] S14: by inverse Hough transform, the detected parameters are converted back to the actual line positions in the feature space, the book area is detected, and then the semantic lines are output;

[0018] S15: Obtain the coordinates of four corner points according to the output semantic line.

[0019] Specifically, S2: performing geometric transformation on the irregular quadrilateral book area defined by the coordinates of the four corner points through a perspective transformation module, including:

[0020] S21: Apply the following perspective transformation formula to find the values ​​of X and Y;

[0021]

[0022] According to the values ​​of X and Y, use the following formula to find the values ​​of x and y:

[0023]

[0024] Among them, (x, y) is the pixel coordinate of the corrected front view, (u, v) is the pixel coordinate of the distorted image, and a 11 ,a 12 ,a 13 ,a 21 ,a22 ,a 23 ,a 31 ,a 32 ,a 33 are the parameters of the perspective transformation.

[0025] Specifically, in S3, the improved UNet network includes:

[0026] S31: Use VGG16 as the backbone network of UNet, improve the efficiency of feature extraction through transfer learning, and enhance the generalization ability of the model;

[0027] S32: Introducing Haar wavelet downsampling to replace the traditional maximum pooling operation to speed up network convergence while retaining more image details;

[0028] S33: Use CBAM dual attention mechanism to optimize feature extraction in channel and spatial dimensions to further improve segmentation accuracy;

[0029] S34: DySample dynamic upsampling is used to replace the original bilinear interpolation method to improve the accuracy of image reconstruction and reduce memory usage.

[0030] Specifically, the Haar wavelet downsampling HWD:

[0031] The HWD module includes a lossless feature encoding block and a feature learning block;

[0032] In the lossless feature coding block, the feature map is transformed and its spatial resolution is reduced by applying the Haar wavelet transform; the image with a resolution of H×W is decomposed, and the approximate information and high-frequency information are extracted by using high-pass H0 and low-pass H1 decomposition filters; by performing a downsampling operation (↓2) of the approximate and detail components, the approximate low-frequency (A) and detail high-frequency components in the horizontal (H), vertical (V) and diagonal (D) directions are generated. The basis function and scale function of the Haar wavelet transform are mathematically expressed as:

[0033]

[0034] Among them, j and k represent the order of basis function;

[0035] The feature learning block consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer, which is used to adjust the number of channels of the feature map and filter out redundant information as much as possible.

[0036] Specifically, the CBAM dual attention mechanism:

[0037] The CBAM module includes a channel attention submodule and a spatial attention submodule, which adaptively optimize features from the channel dimension and spatial dimension respectively;

[0038] The calculation of the channel attention part is shown in formula (6); input feature map F∈R W×H×C Perform global average pooling and maximum pooling operations simultaneously;

[0039] The two sets of results are fed into a multi-layer perceptron (MLP) with shared weights for nonlinear transformation;

[0040] Finally, the output results of MLP are added element by element, and the features are fused and then activated by sigmoid function to generate the final channel feature M. C (F);

[0041] M C (F)=Sigmoid[MLP(AvgPool(F))+MLP(MaxPool(F))] (6)

[0043] Where AvgPool(F) and MaxPool(F) are global average pooling and maximum pooling operations on the input, and MLP stands for multi-layer perceptron;

[0044] The calculation of the spatial attention part is shown in formula (7), where the channel feature M C The feature map F′ obtained by multiplying (F) by the original input F is used as the input of the spatial attention module. F′ performs global average pooling and maximum pooling operations along the channel dimension, then performs channel dimension splicing, and then undergoes a layer of convolution operation. Finally, the spatial feature M is generated by the Sigmoid activation function. S (F′), and finally multiply the generated spatial features by F′ to obtain the feature map finally generated by the CBAM module:

[0045]

[0046] In the formula, f 7×7 Indicates that the convolution kernel size is 7×7.

[0047] Specifically, in the DySample network structure, given a feature map X∈R W×H×C After the linear layer and pixel shuffle operation, the offset O∈R is generated sW×sH×2g , and then the generated offset and the original grid volume G∈R sW×sH×2g Add together to get the sampling set S∈R sW×sH×2g ; Finally, the sampling set S upsamples the input feature map X to X′ through the grid_sample function in PyTorch. The above process is expressed by the following equation:

[0048] O=0.5·σ(linear1(X)·linear2(X)) (8)

[0049] S=G+O (9)

[0050] X′=grid_sample(X,S) (10).

[0051] The present invention provides a book correction and segmentation method suitable for smart libraries. The method corrects the perspective distortion generated by the camera during image acquisition and combines the improved UNet algorithm to segment the corrected book image. It can not only effectively correct the image distortion caused by the change of shooting angle, but also show excellent performance in the book spine segmentation task, providing reliable technical support for the automated inventory of smart libraries. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 It is an overall framework of a book correction and segmentation algorithm for an intelligent library of the present invention;

[0055] Figure 2 A schematic diagram of a deep Hough transform network of the present invention;

[0056] Figure 3 It is a schematic diagram of the perspective transformation principle of the present invention;

[0057] Figure 4 This is a schematic diagram of the improved UNet network structure of the present invention;

[0058] Figure 5 The present invention is applied to the method of the present invention and the running result is visualized. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0060] In view of the many problems that the automated inventory technology in the smart library in the prior art still faces, this embodiment provides a book correction and segmentation method applicable to the smart library, including:

[0061] S1: Introduce the deep Hough transform network DHT, the network diagram is as follows Figure 2 As shown in the figure. This network is used to specifically deal with image distortion caused by camera rotation. Through the DHT network, the book area can be accurately detected, key semantic lines can be extracted, and the coordinates of the four corner points of the book's quadrilateral area can be further obtained, laying the foundation for subsequent image correction.

[0062] 1) The input image is processed by a convolutional neural network encoder to extract pixel-level deep features;

[0063] 2) Convert these features from feature space to parameter space through deep Hough transform;

[0064] 3) Processing with a context-aware line detector. By performing convolution operations in parameter space, the candidate lines and their surrounding context features are aggregated and analyzed;

[0065] 4) Through the inverse Hough transform, the detected parameters are converted back to the actual line positions in the feature space, the book area is detected, and then the semantic lines are output;

[0066] 5) Based on the two output semantic lines, the precise coordinates of the four corner points are obtained by calculating their intersections with the image boundary. These corner point coordinates provide the necessary geometric basis for subsequent image correction, thereby ensuring that the true form of the image can be effectively restored during the perspective transformation process.

[0067] S2: Introduce the perspective transformation module to perform geometric transformation on the irregular quadrilateral book area defined by the four corner point coordinates obtained in step 1. Through this transformation, the irregular quadrilateral is adjusted to a regular rectangular area, thereby obtaining a frontal book image without background interference, providing a better image basis for subsequent accurate segmentation. The schematic diagram of the perspective transformation principle is shown in the figure below. Figure 3 shown.

[0068] 1) Number the coordinates of the four corner points 1-4 in clockwise order;

[0069] 2) Apply the following perspective transformation formula to find the X and Y values ​​of each coordinate;

[0070]

[0071] 3) According to the values ​​of X and Y, use the following formula to find the values ​​of x and y:

[0072]

[0073] Among them, (x, y) is the pixel coordinate of the corrected front view, (u, v) is the pixel coordinate of the distorted image, and a 11 ,a 12 ,a 13 ,a 21 ,a 22 ,a 23 ,a 31 ,a 32 ,a 33 are the parameters of the perspective transformation.

[0074] 4) Draw the corrected book image using the obtained front view pixel coordinates and save it.

[0075] S3: Apply the improved UNet network to segment the spine of the book image after perspective correction. The schematic diagram of the improved UNet network is as follows: Figure 4 shown.

[0076] 1) Use VGG16 as the backbone network of UNet, improve the efficiency of feature extraction through transfer learning, and enhance the generalization ability of the model. As a classic deep convolutional neural network, VGG16 has a simple structural design and good feature extraction capabilities. Through a series of stacked 3x3 convolution kernels, VGG16 can capture image features at different scales while keeping the number of parameters relatively controllable. Therefore, using VGG16 to replace the backbone network of UNet can improve the expressiveness of the model while avoiding overly complex network design. Since VGG16 has a deeper network structure, its deep convolution can extract more semantic high-level features, making the model perform better when processing complex images.

[0077] 2) Introducing Haar wavelet downsampling to replace the traditional maximum pooling operation can speed up the network convergence while retaining more image details.

[0078] Specifically, the core structure of the HWD module includes a lossless feature coding block and a feature learning block. In the lossless feature coding block, the feature map is transformed and its spatial resolution is reduced by applying the Haar wavelet transform technique. Specifically, the process decomposes the image with a resolution of H×W, uses high-pass (H0) and low-pass (H1) decomposition filters to extract approximate information and high-frequency information, and then generates approximate low-frequency (A) and detail high-frequency components in the horizontal (H), vertical (V) and diagonal (D) directions by performing a downsampling operation (↓2) of the approximate and detail components, thereby ensuring efficient retention of detail feature information. The basis function and scale function of the Haar wavelet transform can be mathematically expressed as:

[0079]

[0080]

[0081] Where j and k represent the order of the basis function.

[0082] The feature learning block consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer, which is used to adjust the number of channels of the feature map and filter out redundant information as much as possible, so that subsequent layers can learn representative features more efficiently.

[0083] 3) Use the CBAM dual attention mechanism to optimize the feature extraction of channel and spatial dimensions to further improve the segmentation accuracy.

[0084] Specifically, the CBAM module includes two sub-modules: channel attention and spatial attention, which adaptively optimize features from the channel dimension and spatial dimension respectively. The calculation of the channel attention part is shown in formula (6). Input feature map F∈R W ×H×C The global average pooling operation and the maximum pooling operation are performed simultaneously. Then, the two sets of results are sent to the multi-layer perceptron (MLP) with shared weights for nonlinear transformation. Finally, the output results of the MLP are added element by element, and the fused features are passed through the sigmoid activation function to generate the final channel feature M. C (F).

[0085] M C (F)=Sigmoid[MLP(AvgPool(F))+MLP(MaxPool(F))] (6)

[0086] Where AvgPool(F) and MaxPool(F) are global average pooling and maximum pooling operations on the input. MLP stands for Multi-Layer Perceptron.

[0087] The calculation of the spatial attention part is shown in formula (7). C The feature map F′ obtained by multiplying the original input F by (F) is used as the input of the spatial attention module. F′ performs global average pooling and maximum pooling operations along the channel dimension, and then concatenates the channel dimension. Then, it undergoes a layer of convolution operation and finally passes through the Sigmoid activation function to generate the spatial feature M. S (F′). Finally, the generated spatial features are multiplied by F′ to obtain the feature map finally generated by the CBAM module.

[0088]

[0089] In the formula, f 7×7 Indicates that the convolution kernel size is 7×7.

[0090] 4) DySample dynamic upsampling is used to replace the original bilinear interpolation method, which effectively improves the accuracy of image reconstruction while reducing memory usage.

[0091] Specifically, in the DySample network structure, given a feature map X∈R W×H×C After the linear layer and pixel shuffle operation, the offset O∈R is generated sW×sH×2g , and then the generated offset and the original grid volume G∈R sW×sH×2g Add together to get the sampling set S∈R sW×sH×2g Finally, the sampling set S upsamples the input feature map X to X′ through the grid_sample function in PyTorch. This process can be expressed by the following equation:

[0092] O=0.5·σ(linear1(X)·linear2(X)) (8)

[0093] S=G+O (9)

[0094] X′=grid_sample(X,S) (10)

[0095] Experiments show that the book correction and segmentation algorithm for smart libraries proposed in this paper can effectively correct distorted images and accurately segment the spine of the corrected book images, verifying its adaptability and stability in complex scenes. The results of the method are visualized as follows: Figure 5 As shown in Figure 1, the complete processing from distorted image to rectified image and finally to spine segmentation is shown.

[0096] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A book correction and segmentation method suitable for an intelligent library, characterized in that: include: S1: Use the deep Hough transform network to extract the semantic lines of the book area and obtain the coordinates of the four corner points; S2: Through the perspective transformation module, the irregular quadrilateral book area defined by the coordinates of the four corner points is geometrically transformed; the irregular quadrilateral is adjusted to a regular rectangular area, so as to obtain a frontal book image without background interference; S3: Apply the improved UNet network to segment the book spine of the book image obtained in S2.

2. A book correction and segmentation method suitable for an intelligent library according to claim 1, characterized in that: S1 specifically includes: S11: Use a convolutional neural network encoder to process the input image and extract pixel-level deep features; S12: Convert pixel-level deep features from feature space to parameter space through deep Hough transform; S13: Processing using a context-aware line detector, performing a convolution operation in the parameter space, aggregating and analyzing the context features of the candidate lines and their surroundings, and detecting the parameters; S14: by inverse Hough transform, the detected parameters are converted back to the actual line positions in the feature space, the book area is detected, and then the semantic lines are output; S15: Obtain the coordinates of four corner points according to the output semantic line.

3. A book correction and segmentation method suitable for an intelligent library according to claim 1, characterized in that: S2: Through the perspective transformation module, the irregular quadrilateral book area defined by the coordinates of the four corner points is geometrically transformed, including: S21: Apply the following perspective transformation formula to find the values ​​of X and Y; According to the values ​​of X and Y, use the following formula to find the values ​​of x and y: Among them, (x, y) is the pixel coordinate of the corrected front view, (u, v) is the pixel coordinate of the distorted image, and a 11 ,a 12 ,a 13 ,a 21 ,a 22 ,a 23 ,a 31 ,a 32 ,a 33 are the parameters of the perspective transformation.

4. A book correction and segmentation method suitable for an intelligent library according to claim 1, characterized in that: In S3, the improved UNet network includes: S31: Use VGG16 as the backbone network of UNet, improve the efficiency of feature extraction through transfer learning, and enhance the generalization ability of the model; S32: Introducing Haar wavelet downsampling to replace the traditional maximum pooling operation to speed up network convergence while retaining more image details; S33: Use CBAM dual attention mechanism to optimize feature extraction in channel and spatial dimensions to further improve segmentation accuracy; S34: DySample dynamic upsampling is used to replace the original bilinear interpolation method to improve the accuracy of image reconstruction and reduce memory usage.

5. A book correction and segmentation method suitable for an intelligent library according to claim 4, characterized in that: The Haar wavelet downsampling HWD: The HWD module includes a lossless feature encoding block and a feature learning block; In the lossless feature coding block, the feature map is transformed and its spatial resolution is reduced by applying the Haar wavelet transform; the image with a resolution of H×W is decomposed, and the approximate information and high-frequency information are extracted by using high-pass H0 and low-pass H1 decomposition filters; by performing a downsampling operation (↓2) of the approximate and detail components, the approximate low-frequency (A) and detail high-frequency components in the horizontal (H), vertical (V) and diagonal (D) directions are generated. The basis function and scale function of the Haar wavelet transform are mathematically expressed as: Among them, j and k represent the order of basis function; The feature learning block consists of a convolutional layer, a batch normalization layer, and a ReLU activation layer, which is used to adjust the number of channels of the feature map and filter out redundant information as much as possible.

6. A book correction and segmentation method suitable for an intelligent library according to claim 4, characterized in that: The CBAM dual attention mechanism: The CBAM module includes a channel attention submodule and a spatial attention submodule, which adaptively optimize features from the channel dimension and spatial dimension respectively; The calculation of the channel attention part is shown in formula (6); input feature map F∈R W×H×C Perform global average pooling and maximum pooling operations simultaneously; The two sets of results are fed into a multi-layer perceptron (MLP) with shared weights for nonlinear transformation; Finally, the output results of MLP are added element by element, and the features are fused and then activated by sigmoid function to generate the final channel feature M. C (F); M C (F)=Sigmoid[MLP(AvgPool(F))+MLP(MaxPool(F))] (6) Where AvgPool(F) and MaxPool(F) are global average pooling and maximum pooling operations on the input, and MLP stands for multi-layer perceptron; The calculation of the spatial attention part is shown in formula (7), where the channel feature M C The feature map F′ obtained by multiplying (F) by the original input F is used as the input of the spatial attention module. F′ performs global average pooling and maximum pooling operations along the channel dimension, then performs channel dimension splicing, and then undergoes a layer of convolution operation. Finally, the spatial feature M is generated by the Sigmoid activation function. S (F′), and finally multiply the generated spatial features by F′ to obtain the feature map finally generated by the CBAM module: M S (F′)=Sigmoid[f 7×7 [(AvgPool(F));MLP(MaxPool(F))]] (7) In the formula, f 7×7 Indicates that the convolution kernel size is 7×7.

7. The method for book correction and segmentation suitable for intelligent library according to claim 4, characterized in that: In the DySample network structure, given a feature map X∈R W×H×C After the linear layer and pixel shuffle operation, the offset O∈R is generated sW×sH×2g , and then the generated offset and the original grid volume G∈R sW×sH×2g Add together to get the sampling set S∈R sW×sH×2g ; Finally, the sampling set S upsamples the input feature map X to X′ through the grid_sample function in PyTorch. The above process is expressed by the following equation: O=0.5·σ(linear1(X)·linear2(X)) (8) S=G+O (9) X′=grid_sample(X,S) (10).

Images