Spine segmentation and matching method for books in natural placement state

Abstract

The invention discloses a spine segmentation and matching method for books in a natural placement state, and the method comprises the steps: S1, enabling a YOLOv5 network to serve as the front end of a book image, and obtaining a vertical spine image as network input; step 2, a Deeplab v3plus network is used as a spine segmentation network, and a Dense RA-ASPP framework is used to replace an ASPP framework in the Deeplab v3plus network; step 3, adding a CBB module into the Deeplab v3plus network, wherein the CBB module is added into the Deeplab v3plus network; 4, acquiring VGG features, low-frequency features, high-frequency features and high-frequency and low-frequency fusion features of the spine by using VGG16 and FHNet; the method comprises the following steps: step 1, carrying out color clustering on a book spine matching feature library, step 2, carrying out color clustering on the book spine matching feature library, step 3, carrying out color clustering on the book spine matching feature library, step 5, carrying out cascading processing on the color clustering and spine matching feature library to obtain a depth feature cascading matching method, and step 6, segmenting a book image by using Y-Deeplab v3plus to obtain an accurate spine image as a query image of the depth feature cascading matching method, and carrying out depth feature extraction and matching on the query image to obtain a matching result of the spine image. According to the method, the accuracy of spine segmentation and matching can be effectively improved, and the efficiency of spine matching is improved.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular provides a spine segmentation and matching method for books in a naturally placed state. Background Art

[0002] Since the book spine in the book image has good straight line and texture information, most of the traditional book segmentation methods are based on straight line detection and spine texture information to perform book segmentation. N. Tabassum et al. proposed a segmentation method for the spine of multi-line books. This method can use morphological algorithms to remove small area images to obtain accurate spine edges. Cui Chen proposed a spine region positioning method based on text detection combined with support vector machine, which has been improved in terms of illumination sensitivity, adjacent spine color contrast sensitivity, and spine multi-angle tilt detection. Such traditional methods can only be applied to a small number of bookshelf images with thicker and more regular textures. Compared with traditional segmentation algorithms, Zhou et al. proposed a book segmentation and recognition method based on spine depth features. This method uses Mask R-CNN (Mask Region-based Convolutional Neural Network) instance segmentation network to train the spine segmentation model, achieving one-step segmentation and better flexibility and adaptability. Zhang Kexin et al. proposed an improved Deeplab v3 plus book spine segmentation algorithm, which introduced denser pyramid pooling layers and strip pooling layers to more effectively extract the long strip features of the book spine, providing a solution for the segmentation of book spines in a natural placement state. However, the disadvantage is that it cannot handle the more complex book spine image segmentation problem. In book matching, most matching algorithms are based on feature descriptors for book matching. Fowers et al. proposed a new feature descriptor called Color Difference of Gaussians SIFT (CDoG-SIFT), which can well avoid the problem of unclear edges of grayscale image targets in previous studies. By extracting this feature from the book spine image and matching it with the book spine feature database, it can identify misplaced or lost books. Cui Chen used SIFT features, approximate nearest neighbor algorithm and random sampling consensus (RANSAC) feature matching algorithm to design a multi-target template matching method for matching book spine targets in bookcase scene images. It has a good matching effect on book spines in vertically densely arranged, tilted placement, and different lighting conditions.

[0003] In general, the existing book spine segmentation algorithms are not very adaptable and can only be used in scenes where books are neatly placed. It is difficult to effectively segment books that are tilted at large angles or have a variety of placement states in the image. At the same time, the existing book spine matching algorithms have the problem of low matching efficiency. When the target is to match tens of thousands or hundreds of thousands of data, the running speed is slow, resulting in the fact that the spine segmentation and matching method of books on the shelves in a natural placement state in the library is still a challenging research topic.

[0004] In summary, it is very meaningful to provide a spine segmentation and matching method suitable for books in a naturally placed state. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a spine segmentation and matching method for books in a naturally placed state, so as to solve the problems of spine segmentation of books on the shelves in a naturally placed state in a library, and poor real-time performance and low accuracy in matching large amounts of books.

[0006] The technical solution provided by the present invention is: a method for segmenting and matching the spines of books in a naturally placed state, comprising:

[0007] S1: Use the YOLOv5 network as the front end of the book image to process the tilted spine image in the image, perform affine transformation, and obtain the vertical spine image as the network input;

[0008] S2: Use the Deeplab v3 plus network as the spine segmentation network and MobileNet V2 as the feature extraction network. Based on the Deeplab v3 plus network, replace the ASPP framework in Deeplab v3 plus with the Dense RA-ASPP framework.

[0009] S3: The shallow features are fused with the deep features to obtain the spine features with more comprehensive semantic information. The CBB module is added to the Deeplab v3plus network to obtain the Y-Deeplab v3 plus spine segmentation network.

[0010] S4: Use VGG16 and FHNet to obtain the VGG features, low-frequency features, high-frequency features, and high-frequency and low-frequency fusion features of the spine, and build a feature library for spine matching;

[0011] S5: Perform color clustering on all book spine images, and cascade the color clustering with the book spine matching feature library to obtain a deep feature cascade matching method.

[0012] S6: Use Y-Deeplab v3 plus to segment the book image and obtain an accurate spine image as the query image for the deep feature cascade matching method. Perform deep feature extraction and matching on the query image to obtain the matching result of the spine image.

[0013] Preferably, in S1, the YOLOv5 network is used as the book image front end to process the tilted book spine image in the image, including:

[0014] S11: The book image is divided into three states: vertical state, tilted state and mixed state; the YOLOv5 network is used to detect the tilted book and the tilted part of the book in the mixed state;

[0015] S12: obtaining the edge of the spine image by edge detection of the book in the tilted state and the tilted part of the book in the mixed state, and obtaining a clear spine edge;

[0016] S23: Using Hough line detection to detect the book spine gap, obtain the inclined gap lines of multiple book spines, calculate the angle between the gap line and the horizontal line, and calculate the average value of the angle;

[0017] S24: Using the average value as the transformation angle of affine transformation, the tilted spine image is transformed into a vertical state through radial transformation.

[0018] Preferably, the Dense RA-ASPP module structure in S2 is composed of a strip pooling layer, a residual structure consisting of a depthwise separable asymmetric convolution with void rates of 3, 6, 12, 18, and 24, and a 1×1 convolution, respectively, wherein the depthwise separable asymmetric convolution is composed of a superposition of a 3×3 convolution, a 1×3 convolution, and a 3×1 convolution;

[0019] The Dense RA-ASPP module concatenates the features obtained after each layer of depthwise separable asymmetric convolution with the final output features of MobileNetV2 and sends them to the next layer: after adjusting the number of channels through 1×1 convolution, the self-attention mechanism is used to further strengthen its own features, and then the deep features are obtained through upsampling operation. The deep features are concatenated with the shallow features that have not passed through the Dense RA-ASPP module to obtain the spine features with more comprehensive semantic information.

[0020] Preferably, in S3, the CBB module is composed of a bottleneck block consisting of two branches, one of which is subjected to 3×3 convolution and SE attention module to further extract more detailed feature information, and the other branch is subjected to 1×1 convolution to adjust the number of channels after the deep features and the shallow features are spliced, retain their feature information, and fuse the features of the two branches to form an improved Deeplab v3 plus network.

[0021] Preferably, the FHNet is used to extract low-frequency features, high-frequency features, and high-frequency and low-frequency fusion features of the spine:

[0022] The backbone network of the FHNet is ResNet50, which divides the features into low-frequency features and high-frequency features through the pooling separation module, adds a self-attention mechanism to the high-frequency feature part to strengthen the high-frequency feature information, and then fuses it with the low-frequency feature to obtain high-frequency and low-frequency fusion features;

[0023] The VGG16 feature extraction network is used to extract the VGG features of the spine image. The low-frequency features, high-frequency features, high-frequency and low-frequency fusion features, and VGG features of the spine are used to construct a feature library for spine matching.

[0024] The present invention provides a spine segmentation and matching method for books in a naturally placed state. The method can effectively handle book detection and book segmentation problems in various states, solve the problem of poor real-time matching of books with large amounts of data, and can effectively improve the accuracy of spine segmentation and matching, greatly improve the efficiency of spine matching, and effectively provide favorable guarantees for the inventory of books in unmanned libraries. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments:

[0026] Figure 1 A flowchart of the overall framework of a method for segmenting and matching the spine of a book in a naturally placed state provided by an embodiment of the present invention;

[0027] Figure 2 The network structure diagram of Y-Deeplab v3 plus provided for the disclosed embodiment of the present invention;

[0028] Figure 3 The improved Deeplab v3 plus network structure diagram provided for the disclosed embodiment of the present invention;

[0029] Figure 4 A structural diagram of a deep feature cascade matching method provided in an embodiment disclosed in the present invention;

[0030] Figure 5 A comparison diagram of the book affine transformation results and a semantic segmentation effect diagram of the spine image provided in the disclosed embodiment of the present invention;

[0031] Figure 6 A diagram showing the deep feature cascade matching results provided in the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described in detail below.

[0033] A method for segmenting and matching the spines of books placed naturally. The overall framework flow chart is as follows: Figure 1 Specifically, the method comprises the following steps:

[0034] S1: Use the YOLOv5 network as the front end of the book image to process the tilted spine image in the image, perform affine transformation, and obtain the vertical spine image as the network input;

[0035] YOLOv5 is used to detect the partition state of the book spine in the book image. The inclined area of ​​the spine is vertically cut according to the Y value corresponding to the detection box to obtain the inclined area of ​​the spine. Then, Hough line detection is used to obtain multiple gap lines of the spine in the inclined area. The angle between the inclined gap line and the horizontal line is calculated, and the average value is taken as the transformation angle of the affine transformation. After that, an affine transformation is performed to transform it into an image with a vertical spine as the input of the spine segmentation network model.

[0036] Specifically, 1) first, the book image is divided into three states: vertical state, tilted state and mixed state; the YOLOv5 network is used to detect the tilted book and the tilted part in the mixed state, and the Canny edge detection is used to obtain the edge of the spine image to obtain a clear spine edge;

[0037] 2) Use Hough line detection to detect the gap of the book spine, and get the accurate straight line by setting the threshold value and the minimum straight line length min_line_length in CV2.HoughLines();

[0038] 3) Perform angle statistics based on all the straight lines detected above, and take the average of the statistical angles to obtain the uniform tilt angle of the book in the image, that is, the affine transformation angle θ Aver ;

[0039] 4) Affine transformation takes the center of the image as the origin and moves the center of the image to the origin of the coordinate system through translation transformation. The translation transformation matrix T1 is shown in formula 1, where (c x , c y ) is the center point of the image. Then, an affine transformation is performed, where the rotation matrix R (clockwise rotation is positive) is shown in Formula 4, where θ is the straight line angle calculated by Hough line detection. Since no scaling transformation is performed, the scaling factor is 1, and the transformed image is obtained;

[0040]

[0041] The transformed image is translated to the original position, that is, the inverse transformation of the translation T2 = T1 -1 , get the final affine transformation matrix T AF=T2 R T1, multiplied by the original image, we get the image after affine transformation, that is, transforming the tilted spine image into a vertical state.

[0042] Step 2: Use the Deeplab v3 plus network as the spine segmentation network and MobileNet V2 as the feature extraction network. The shallow features are fused with the deep features to obtain the spine features with more comprehensive semantic information.

[0043] The ASPP module in Deeplab v3 plus is replaced with the Dense RA-ASPP module, and the original 3×3 convolution is replaced with the depthwise separable asymmetric convolution. The depthwise separable asymmetric convolution is composed of 3×3 convolution, 1×3 convolution and 3×1 convolution, which can extract local features while retaining the long strip characteristics of the spine. In addition, the residual structure is introduced to extract richer semantic information.

[0044] The spine segmentation network uses Deeplab v3 plus as the basic framework and MobileNet V2 as the feature extraction network. First, the book image is input for feature extraction, and the shallow features are fused through upsampling. In the encoding stage, for densely placed book spine images, the Dense RA-ASPP module is introduced to replace the ASPP module. Its structure consists of a strip pooling layer, a depth-separable asymmetric convolution (Dense Atrous Asymmetric Convolution Block, DAACB) with a void rate of 3, 6, 12, 18, and 24, and a residual structure composed of 1×1 convolution. The DAACB structure uses parallel asymmetric convolution (Asymmetric Convolutional Block, ACB). Asymmetric convolution includes convolution kernels in both horizontal and vertical directions, which are 1×3 convolution and 3×1 convolution respectively. This design can better capture the horizontal and vertical features in the image, and perform feature fusion through the additivity of convolution, which helps to improve the performance of the model.

[0045] Compared with the original ASPP module, the improved new module adopts a denser feature fusion method. Each layer uses different void rates to extract spine features with different receptive fields, and adopts a residual network structure in each layer. Secondly, Dense RA-ASPP concatenates the features obtained after depthwise separable asymmetric convolution of each layer with the final output features of MobileNetV2 and sends them to the next layer, so that the final output contains feature maps with multiple receptive fields and multi-scales.

[0046] Step 3: Add the CBB module to the Deeplab v3 plus network to obtain the Y-Deeplab v3 plus spine segmentation network;

[0047] After the shallow features are fused with the deep features, the CBB module is added to further extract feature information and reduce information loss during network transmission;

[0048] Specifically,

[0049] 1) In the decoding stage, after passing through the Dense RA-ASPP module, the number of channels is adjusted through 1×1 convolution, and then the self-attention mechanism is used to further strengthen its own features. Finally, the deep features are obtained through upsampling operation. The deep features are concatenated with the shallow features that have not passed through the Dense RA-ASPP module to obtain the spine features with more comprehensive semantic information.

[0050] 2) Since the Deeplab v3 plus network uses bilinear upsampling operations to obtain segmentation results in the decoding and fusion parts, it only considers the size of adjacent pixel values, but does not consider the influence of the change rate of adjacent points, which may cause some detail information to be lost after magnification. In order to alleviate this problem, the CBB module is introduced to replace the traditional 3×3 convolution module, such as Figure 3 As shown in the figure, the CBB module is a parallel fusion structure consisting of a 1×1 convolution and a bottleneck block. One branch adjusts the channel and resolution by adding a 1×1 convolution, and adds an SE attention module after the 3×3 convolution, reducing the factor to 16 to further extract more detailed feature information. The other branch uses a 1×1 convolution to only adjust the number of channels after the deep features and shallow features are concatenated, retaining their feature information and fusing the features of the two branches. Before upsampling, feature information is further extracted to reduce information loss during network transmission, forming an improved Deeplab v3 plus network.

[0051] YOLOv5 is used as its front end, and the Dense RA-ASPP module replaces the ASPP module in Deeplab v3 plus. After the shallow features are fused with the deep features, the CBB module is added to obtain the Y-Deeplab v3 plus book spine segmentation network.

[0052] Specifically include:

[0053] 1) Using YOLOv5 as its front end, the state detection of the book spine in the book image and affine transformation are performed to obtain a better segmentation network input image.

[0054] 2) The Dense RA-ASPP module replaces the ASPP module in Deeplab v3 plus, so that the final spine feature has multi-receptive field and multi-scale information.

[0055] 3) After the shallow features are fused with the deep features, the CBB module is added to obtain the Y-Deeplab v3 plus book spine segmentation network.

[0056] 4) The mean intersection over union (MIoU) is used as the evaluation index of the segmentation effect, as shown in formula 3:

[0057]

[0058] Where N is the number of samples, IoU_i (intersection over union of the i-th sample) refers to the ratio of the intersection area to the union area between the predicted result of the i-th sample and the true label, and MIoU represents the average intersection over union of the predicted area and the manually labeled area. For each sample i, use formula 4 to calculate the intersection over union, where is the area of ​​the intersection between the predicted result and the true label, is the area of ​​their union.

[0059] Step 4: Use VGG16 and the newly proposed FHNet to obtain the VGG features, low-frequency features, high-frequency features, and high-frequency and low-frequency fusion features of the spine. The proposed FHNet can simultaneously extract the low-frequency features, high-frequency features, and high-frequency and low-frequency fusion features of the spine. Its backbone network is ResNet50. The features are divided into low-frequency features and high-frequency features through the pooling separation module. The self-attention mechanism is added to the high-frequency feature part to strengthen the high-frequency feature information, and then it is fused with the low-frequency features to obtain the high-frequency and low-frequency fusion features, and a feature library for spine matching is constructed. The specific process of the proposed FHNet is as follows:

[0060] 1) FHNet can simultaneously extract low-frequency features, high-frequency features, and high-frequency and low-frequency fusion features of images. First, ResNet50 is used as the backbone network to obtain the feature F of the spine image;

[0061] 2) The obtained spine features are sent to the pooling separation module, where the pooling separation module is divided into an average pooling module and a maximum pooling module. According to formula (5), after pooling, the low-frequency features FL of the spine image are obtained, and these features are returned to the original size after upsampling. According to formula (6), the spine features F are used to subtract the upsampled low-frequency features to obtain the high-frequency features FH of the spine image.

[0062] FL=Pool(F) (5)

[0063] FH=F-Upsample(FL) (6)

[0064] 3) The obtained high-frequency feature FH is passed through the self-attention mechanism to strengthen the high-frequency feature information, and then it is fused with the low-frequency feature through the feature fusion module to obtain the high-frequency and low-frequency fusion feature F1, where the feature fusion module is a parallel block composed of 3×3 convolution and 3 3×3 convolutions.

[0065] 4) As in the previous two steps, the obtained high-frequency and low-frequency fusion features F1 are used as the input of the maximum pooling separation module to obtain the low-frequency features F1L, high-frequency features F1H1, and high-frequency and low-frequency fusion features F2 of the final spine image. The VGG16 feature extraction network is then used to extract the VGG features of the spine image, and these features are used to construct a feature library for spine matching.

[0066] Step 5: All book spine images are clustered by color, and the book spine image library is divided into 10 categories according to color to reduce the amount of search data. Then, low-frequency features, high-frequency features, high-frequency and low-frequency fusion features, and VGG deep features are extracted through FHNet and VGG16 in turn, and cascaded to obtain the deep feature cascade matching method.

[0067] 1) Perform color clustering on the 2025 book spine images in the example. First, convert the color space of the book spine image to the HSV space, and perform brightness equalization on the brightness channel V to reduce the impact of light intensity on the image. Perform statistics on its color histogram features, take the maximum value in the histogram as its main color feature, and perform average color histogram statistics in the color space to obtain its average color feature. The average color feature is fused with the main color feature of the HSV space to obtain the color feature of the book spine image, and then perform Kmeans clustering with 10 categories, and calculate and retain the cluster center of each category to reduce the magnitude of the search data.

[0068] 2) Cluster the query image according to the previous step, and calculate the similarity with the 10 cluster centers, divide it into one of the 10 categories, and obtain the feature library corresponding to the cluster center (first-level matching retrieval library); then extract the low-frequency features of the query image through FHNet, calculate the similarity of the low-frequency features, and take the top 100 spine images with similarity to obtain its second-level matching retrieval library; then extract the high-frequency features of the query image, calculate the similarity of the high-frequency features, and take the top 50 spine images with similarity to obtain its third-level matching retrieval library; then extract the high-frequency and low-frequency fusion features of the query image, calculate the similarity of the high-frequency and low-frequency fusion features, and take the top 25 spine images with similarity to obtain its third-level matching retrieval library; then extract the VGG features of the query image, calculate the similarity of the VGG features, and take the spine image with the largest similarity to obtain the deep feature cascade matching method in the form of a cascade.

[0069] Step 6: Use Y-Deeplab v3 plus to segment the book spine and obtain an accurate spine image. Then use it as the query image for the deep feature cascade matching method, extract deep features from the query image, and perform similarity calculations between the query spine features and the feature library to obtain the matching results of the spine image.

[0070] Experiments have shown that the Y-Deeplab v3 plus network model proposed in this invention can effectively solve the problem of book spine segmentation in the natural state of libraries in real life, especially in dealing with tilted books and mixed state (vertical and tilted) books, which has great advantages over other algorithms. At the same time, in the test of 2025 book spine images, the accuracy of the deep feature cascade matching method is 97.97%, and the matching retrieval time for each book spine is about 0.3s, which proves the effectiveness of the method of this invention for matching large amounts of books.

[0071] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for segmenting and matching the spines of books placed naturally, characterized in that: include: S1: Use the YOLOv5 network as the front end of the book image to process the tilted spine image in the image, perform affine transformation, and obtain the vertical spine image as the network input; S2: Use the Deeplab v3 plus network as the spine segmentation network and MobileNet V2 as the feature extraction network. Based on the Deeplab v3 plus network, replace the ASPP framework in Deeplab v3 plus with the Dense RA-ASPP framework. S3: The shallow features are fused with the deep features to obtain the spine features with more comprehensive semantic information. The CBB module is added to the Deeplab v3 plus network to obtain the Y-Deeplab v3 plus spine segmentation network. S4: Use VGG16 and FHNet to obtain the VGG features, low-frequency features, high-frequency features, and high-frequency and low-frequency fusion features of the spine, and build a feature library for spine matching; S5: Perform color clustering on all book spine images, and cascade the color clustering with the feature library of book spine matching to obtain a deep feature cascade matching method; S6: Use Y-Deeplab v3 plus to segment the book image and obtain an accurate spine image as the query image for the deep feature cascade matching method. Perform deep feature extraction and matching on the query image to obtain the matching result of the spine image.

2. The spine segmentation and matching method of a book in a naturally placed state according to claim 1, characterized in that: In S1, the YOLOv5 network is used as the front end of the book image to process the tilted spine image in the image, including: S11: The book image is divided into three states: vertical state, tilted state and mixed state; the YOLOv5 network is used to detect the tilted book and the tilted part of the book in the mixed state; S12: obtaining the edge of the spine image by edge detection of the book in the tilted state and the tilted part of the book in the mixed state, and obtaining a clear spine edge; S23: Using Hough line detection to detect the book spine gap, obtain the inclined gap lines of multiple book spines, calculate the angle between the gap line and the horizontal line, and calculate the average value of the angle; S24: Using the average value as the transformation angle of affine transformation, the tilted spine image is transformed into a vertical state through radial transformation.

3. The spine segmentation and matching method of a book in a naturally placed state according to claim 1, characterized in that: The Dense RA-ASPP module structure in S2 is composed of a strip pooling layer, a residual structure consisting of a depthwise separable asymmetric convolution with a void rate of 3, 6, 12, 18, and 24, and a 1×1 convolution. The depthwise separable asymmetric convolution is composed of a superposition of a 3×3 convolution, a 1×3 convolution, and a 3×1 convolution. The Dense RA-ASPP module concatenates the features obtained after each layer of depthwise separable asymmetric convolution with the final output features of MobileNetV2 and sends them to the next layer: after adjusting the number of channels through 1×1 convolution, the self-attention mechanism is used to further strengthen its own features, and then the deep features are obtained through upsampling operation. The deep features are concatenated with the shallow features that have not passed through the Dense RA-ASPP module to obtain the spine features with more comprehensive semantic information.

4. The spine segmentation and matching method of a book in a naturally placed state according to claim 1, characterized in that: In S3, the CBB module is composed of a bottleneck block consisting of two branches, one of which is further extracted with more detailed feature information through 3×3 convolution and SE attention module, and the other is adjusted through 1×1 convolution to retain the feature information after the deep features and shallow features are concatenated. The features of the two branches are fused to form an improved Deeplab v3plus network.

5. The spine segmentation and matching method of a book in a naturally placed state according to claim 1, characterized in that: The FHNet is used to extract the low-frequency features, high-frequency features and high-frequency and low-frequency fusion features of the spine: The backbone network of the FHNet is ResNet50, which divides the features into low-frequency features and high-frequency features through the pooling separation module, adds a self-attention mechanism to the high-frequency feature part to strengthen the high-frequency feature information, and then fuses it with the low-frequency feature to obtain high-frequency and low-frequency fusion features; The VGG16 feature extraction network is used to extract the VGG features of the spine image. The low-frequency features, high-frequency features, high-frequency and low-frequency fusion features, and VGG features of the spine are used to construct a feature library for spine matching.

Images