Spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation

Abstract

The invention provides a spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation, and the method comprises the steps: a main branch employs MobileNetV2 as a feature extraction network, and extracts the semantic features of a spine image; the extracted features are sent to a feature separation module, and the high-frequency features and the low-frequency features are fused to obtain spine features; the method comprises the following steps of: performing frequency domain enhancement processing on a spine image by using a dynamic spectrum transformation (DST) module to obtain an improved spine image feature; splicing the high-frequency feature, the low-frequency feature, the high-low frequency fusion feature and the improved spine feature to obtain a spine feature; the method comprises the following steps of: introducing a deformable feature attention module (DFAM) to obtain a shallow layer feature and a deep layer feature which have the capability of capturing a slender object; and step 6, combining the dynamic spectrum transformation branch with the main branch, constructing a multi-branch feature fusion segmentation network, and further fusing shallow features and deep features, thereby improving segmentation precision and adaptability to complex scenes.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a spine segmentation method based on multi-branch feature fusion based on dynamic spectrum transformation. Background Art

[0002] At present, book segmentation methods are mainly divided into methods based on traditional book features and methods based on deep learning. Traditional algorithms usually rely on the straight line and texture information of the spine in the book image to achieve segmentation. N. Tabassum 0 et al. proposed a method for segmenting the spines of multi-line books. This method can use morphological algorithms to remove small area images to obtain accurate spine edges. Cui Chen0 proposed a spine area positioning method based on text detection and support vector machine, which improved lighting sensitivity, adjacent spine color contrast sensitivity, and spine multi-angle tilt detection. Such traditional methods have limited applicability and are mostly only applicable to a small number of thick bookshelf images with regular textures. They cannot handle complex book placement scenes. In contrast, the spine segmentation method based on deep learning has stronger adaptability. Zhou 0 et al. proposed a book segmentation and recognition method based on the deep features of the book spine. This method uses the Mask R-CNN (Mask Region-based Convolutional Neural Network) instance segmentation network to train the book spine segmentation model, achieving one-step segmentation and better flexibility and adaptability. 0et al. proposed an improved Deeplab v3plus book spine segmentation algorithm, which introduced denser pyramid pooling layers and strip pooling layers to more effectively extract the long strip features of the book spine, providing a solution for the segmentation of book spines in natural placement. However, the disadvantage is that it cannot handle more complex book spine image segmentation problems, and does not effectively utilize the texture, contour, and edge information in the book spine image, resulting in poor spine segmentation results in practical applications. In general, the existing book spine segmentation algorithms are not very adaptable, and the network has weak detail capture and anti-interference capabilities. They can only be used in scenes where books are neatly placed, and it is difficult to effectively segment irregularly placed books in images. (For details, see [1] N. Tabassum, S. Chowdhury, MK Hossen and SU Mondal, "An approach to recognize book title from multi-cell bookshelf images," 2017 IEEE International Conference on Imaging, Vision & Pattern Recognition (icIVPR), Dhaka, Bangladesh, 2017, pp. 1-6. [2] Cui Chen, Ren Mingwu. A spine location method based on text detection [J]. Computer and Digital Engineering, 2020, 48 (01): 178-182 + 251. (CUI C, REN M WA method of spine location based on text detection [J]. Computer and Digital Engineering, 2020, 48 (01): 178-182 + 251. [3]) Shuo Zhou, Tan Sun, Xue Xia, Library on-shelf book segmentation and recognition based on deep visual features, Information Processing & Management

[0003] ,ISSN 0306-4573,https: / / doi.org / 10.1016 / j.ipm.2022.103101.[4] Ji Xiaofei, Zhang Kexin, Tang Lirong. Book spine segmentation algorithm based on improved Deeplab v3 plus network[J / OL].Journal of Computer Applications: 1-7[2024-02-25].http: / / kns.cnki.net / kcms / detail / 51.1307.TP.20230330.1057.004.html.)

[0004] In summary, in view of the problems and shortcomings of the above-mentioned prior art, it is very meaningful to provide a method for realizing multi-level optimization of book spine segmentation. Summary of the invention

[0005] In view of this, the object of the present invention is to provide a spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation to solve the spine segmentation problem of books on the shelf in the library.

[0006] The technical solution of the present invention is: a spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation, comprising:

[0007] S1: The segmentation network framework adopts an encoding-decoding structure. The main branch of the encoding stage uses MobileNetV2 as the feature extraction network, combined with the dynamic spectrum transformation branch as the encoding part, to perform feature encoding and extract the semantic features of the spine image;

[0008] S2: Send the semantic features of the spine image to the feature separation module to extract high-frequency features and low-frequency features respectively, and fuse the high-frequency features and low-frequency features to obtain high- and low-frequency fusion features;

[0009] S3: Introduce a dynamic spectrum transformation branch, use the dynamic spectrum transformation module DST to perform frequency domain enhancement processing on the spine image, and obtain improved spine image features;

[0010] S4: Combine the high-frequency features, low-frequency features, and high-low-frequency fusion features in S2 with the improved spine image features in S3 to obtain spine features with stronger adaptability and richer semantic information;

[0011] S5: The DFAM module is introduced to extract the low-level features extracted after MobileNetV2 and the spine features obtained by S4, and obtain shallow and deep features capable of capturing slender objects;

[0012] S6: Combine the main branch of S1 with the dynamic spectrum transformation branch of S3 to construct a segmentation network that fuses dynamic spectrum multi-branch features, and further fuse the shallow features and deep features obtained in S5 to obtain the spine segmentation result.

[0013] Specifically, the encoding-decoding structure described in S1 is composed of the following: the main branch MobileNetV2 feature extraction network and the dynamic spectrum transformation branch are the encoding part; the concatenation, 3×3 convolution and upsampling operations are the decoding part.

[0014] Specifically, in S2, the features extracted by MobileNetV2 are sent to the feature separation module, and the features are divided into low-frequency features and high-frequency features through the pooling separation module. A self-attention mechanism is added to the high-frequency feature part to strengthen the high-frequency feature information. At the same time, the enhanced high-frequency features are fused at multiple scales and then fused with the low-frequency features to obtain high-frequency and low-frequency fused features.

[0015] Specifically, in S3, the input book image is cut into pieces, and feature encoding is performed, and normalization and two-dimensional Fourier transform 2D FFT are performed in sequence to obtain the features of the spine image in the frequency domain;

[0016] The characteristics of the spine image in the frequency domain are sent to the dynamic spectrum transformation module DST to reconstruct the low-frequency characteristics and introduce random interference to obtain adaptively enhanced spectrum characteristics;

[0017] The adaptively enhanced spectral features are sequentially subjected to inverse Fourier transform (2D IFFT) and normalized to obtain improved book spine image features.

[0018] Specifically, in S5, the DFAM module is introduced into the segmentation network. First, in the main branch feature extraction stage, low-level features of different scales are upsampled and concatenated to obtain multi-scale low-level features. The obtained low-level features are sent to the DFAM module after 1×1 convolution to enhance the ability to capture slender objects and obtain shallow features of the spine image.

[0019] The book spine features spliced ​​in S4 are subjected to 1×1 convolution and DFAM modules to enhance the ability to capture slender objects. The deep features of the book spine image are obtained through upsampling operations.

[0020] The present invention provides a spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation, which realizes multi-level optimization of spine segmentation by combining DFAM module. The main branch is combined with the dynamic spectrum transformation branch, and shallow and deep features are integrated, so that the segmentation network can not only flexibly adapt to multi-angle and slender objects in complex scenes, but also use the adaptive convolution and attention mechanism of DFAM to enhance the ability to capture slender or irregular spines. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 An overall framework diagram of a spine segmentation method based on multi-branch feature fusion based on dynamic spectrum transformation provided by the disclosed embodiment of the present invention;

[0024] Figure 2 A structural diagram of a pooling separation module provided in an embodiment disclosed in the present invention;

[0025] Figure 3 A structural diagram of a feature fusion module provided in the disclosed embodiment of the present invention;

[0026] Figure 4 A structural diagram of a dynamic spectrum conversion module provided in an embodiment disclosed in the present invention;

[0027] Figure 5 A structural diagram of a deformable feature attention module provided for an embodiment disclosed in the present invention. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0029] In view of the fact that the existing technology cannot solve the problem of book segmentation in a natural placement state in an unmanned library based on computer vision, this embodiment provides a spine segmentation method based on multi-branch feature fusion based on dynamic spectrum transformation, including:

[0030] S1: The segmentation network framework adopts an encoding-decoding structure. The main branch of the encoding stage uses MobileNetV2 as the feature extraction network, combined with the dynamic spectrum transformation branch as the encoding part, to perform feature encoding and extract the semantic features of the spine image;

[0031] Using the encoding-decoding structure, the main branch MobileNetV2 feature extraction network and the dynamic spectrum transformation branch are used as the encoding part to perform feature encoding. At the same time, concatenation, 3×3 convolution and upsampling operations are used as the decoding part to form the encoding-decoding structure.

[0032] Specifically, they may include the following:

[0033] 1) First, the segmentation network framework adopts an encoding-decoding structure. In the encoding stage, the main branch uses MobileNetV2 as the feature extraction network. Secondly, a dynamic spectrum transformation branch is added as the encoding part to perform feature encoding and extract spine features.

[0034] 2) In the decoding stage, 1×1 convolution is used to adjust the number of channels, and upsampling operation is used to adjust the features to the same scale for subsequent splicing and fusion to form a coding-decoding structure.

[0035] S2: Send the semantic features of the spine image to the feature separation module to extract high-frequency features and low-frequency features respectively, and fuse the high-frequency features and low-frequency features to obtain high- and low-frequency fusion features;

[0036] The features extracted by MobileNetV2 are sent to the feature separation module, and the features are divided into low-frequency features and high-frequency features through the pooling separation module. The self-attention mechanism is added to the high-frequency feature part to enhance the high-frequency feature information. At the same time, the enhanced high-frequency features are fused at multiple scales and then fused with the low-frequency features to obtain high-frequency and low-frequency fused features.

[0037] The specific steps include:

[0038] 1) Considering that the spine image contains both low-frequency color texture information and high-frequency detail edge information, the spine features extracted by MobileNetV2 are sent to the feature separation module for feature separation, such as Figure 2 As shown in Figure 2. This module uses average pooling and maximum pooling to obtain the low-frequency features FL of the spine image by reducing the image resolution. At this time, the feature height and width become half of the original size;

[0039] 2) Then use bilinear interpolation to upsample these features back to the original size, and use subtraction, as shown in Formula 1, to subtract the low-frequency feature part of the spine image from the spine feature F extracted by the main branch MobileNetV2 in step 1 to obtain the high-frequency feature FH of the spine image.

[0040] FL=Pooling(F) (1)

[0041] FH=F-Upsample(FL) (2)

[0042] 3) Based on the high-frequency features of the book spine obtained above, a self-attention mechanism is added to further strengthen the high-frequency feature information, so that it pays more attention to the details and edge information of the book spine. At the same time, multi-scale fusion is introduced to upsample and downsample the high-frequency features of the book spine respectively, and obtain high-frequency features with a scale reduction of 1 / 2 times and a scale expansion of 2 times, so that the fused features have different receptive fields.

[0043] 4) The spine low-frequency features and spine multi-scale high-frequency features obtained above are sent to the feature fusion module for feature fusion, such as Figure 3 As shown. This module uses two branches to fuse the spine features separately. First, the high-frequency features are passed through three 3×3Convs to obtain richer high-frequency information. Secondly, the other branch performs 3×3Conv on the low-frequency features to further extract the low-frequency information in the image. Since the low-frequency features are obtained through pooling operations, their size is compressed to 1 / 2 of the original size, so the low-frequency feature dimensions are adjusted to match the high-frequency features through upsampling. Finally, the processed high-frequency features and low-frequency features are spliced, and feature fusion is performed through 3×3Conv and Channel attention in turn. The number of channels is adjusted through 1×1Conv to obtain the final high- and low-frequency fusion features. The process is shown in the following formulas (3), (4), and (5):

[0044] L=Upsample(Conv3×3(FL)) (3)

[0045] H=Conv3×3(Conv3×3(Conv3×3(FH))) (4)

[0046] Out=Conv1×1(CA(Conv3×3(Concatenaet(L,H))) (5)

[0047] S3: Introduce a dynamic spectrum transformation branch, use the dynamic spectrum transformation module DST to perform frequency domain enhancement processing on the spine image, and obtain improved spine image features;

[0048] 1) A dynamic spectrum transformation branch is introduced into the segmentation network. First, the input book image is cut into blocks to generate image blocks and perform feature encoding. Each image block is represented as a feature vector in preparation for spectrum analysis.

[0049] 2) Normalize the image features after segmentation to facilitate subsequent frequency domain conversion, making the features more balanced and reducing amplitude differences.

[0050] 3) The normalized features are sent to the two-dimensional fast Fourier transform module 2D FFT to convert the spatial features of the image into frequency domain features, thereby extracting the frequency information of the image for subsequent spectrum analysis.

[0051] 4) The frequency domain features obtained above are sent to the dynamic spectrum transformation module DST, such as Figure 4 As shown in the figure. First, extract the low-frequency part in the frequency domain features to obtain the structural information of the image. Secondly, reconstruct the low-frequency features to extract effective structural information. Finally, introduce random interference: by adding noise or random sampling, make the spectrum features more robust to enhance the adaptability to interference, thereby improving the anti-interference ability of the features. After reconstruction and random interference processing, the spectrum features with enhanced adaptability are obtained.

[0052] 5) The enhanced spectral features are converted back to the spatial domain through the inverse fast Fourier transform 2D IFFT, so that the features can be applied to subsequent image processing tasks. The inverse transformed features are normalized again to ensure the balanced distribution of feature values ​​to improve the robustness of the model and obtain spine features with certain anti-interference capabilities, providing better input for subsequent feature fusion and classification.

[0053] S4: Combine the high-frequency features, low-frequency features, and high-low-frequency fusion features in S2 with the improved spine image features in S3 to obtain spine features with stronger adaptability and richer semantic information;

[0054] 1) After MobileNetV2 extracts preliminary features, the features are divided into high-frequency and low-frequency features through the feature separation module. High-frequency features mainly contain detailed information such as image texture and edge, reflecting the fine features of the image. Low-frequency features mainly contain the structure and shape information of the image, reflecting the overall outline of the image. High- and low-frequency fusion features fuse high-frequency and low-frequency features, combining the details and structural information of the image, thereby enhancing the expressiveness of the features. The spine features processed by the dynamic spectrum transform branch DST have a certain anti-interference ability.

[0055] 2) Before splicing, use the unsqueeze function to increase the feature dimension of the spine of the spectrum transform branch, and ensure that the feature dimension is consistent with the high-frequency feature, low-frequency feature, and high- and low-frequency fusion feature of the spine obtained in step 2 so that the features can be effectively spliced.

[0056] 3) Normalize the concatenated features to keep the features from different sources consistent in their value ranges and reduce the distribution differences caused by different feature sources.

[0057] 4) The above-mentioned spine high-frequency features, spine low-frequency features, spine high- and low-frequency fusion features and anti-interference features obtained by the dynamic spectrum transformation module DST are spliced ​​in the channel dimension. The final features not only contain high-frequency details and low-frequency structural information, but also have anti-interference properties, thus forming a more adaptable and semantically rich spine feature, providing a more robust feature expression for subsequent segmentation tasks.

[0058] S5: The DFAM module is introduced to extract the low-level features extracted after MobileNetV2 and the spine features obtained by S4, and obtain shallow and deep features capable of capturing slender objects;

[0059] First, in the main branch feature extraction stage, low-level features of different scales are upsampled and spliced ​​to obtain multi-scale low-level features. The obtained low-level features are sent to the DFAM module after 1×1 convolution to enhance the ability to capture slender objects and obtain shallow features of the spine image. Secondly, the spine features spliced ​​in step 4 are subjected to 1×1 convolution and DFAM module to enhance the ability to capture slender objects in the same way, and the deep features of the spine image are obtained after upsampling.

[0060] The details include:

[0061] 1) Introduce the deformable convolutional DFAM module into the segmentation network, where the DFAM module is as follows Figure 5 As shown in the figure. In the DFAM module, the input feature map is first convolved to generate an offset, and the offset is used for deformable convolution to adapt to the shape of slender objects. Next, the spatial attention mechanism is used to generate a spatial weight map through maximum pooling and average pooling, and the deformable feature map is weighted to enhance the spatial attention to the target area. Then, the channel weight vector is generated through channel pooling and a fully connected layer, and the spatially enhanced feature map is channel-weighted to further enhance important information. The final output feature map is enhanced in both spatial and channel dimensions, allowing the network to more effectively capture the features of slender objects.

[0062] 2) First, in the feature extraction stage of the main branch, low-level features of different scales are extracted. These features usually contain image details and edge information. In order to facilitate subsequent feature fusion, the low-level features of different scales are restored to the same spatial resolution through upsampling. The upsampling adopts a bilinear interpolation method to ensure that the feature map is consistent in spatial size. Subsequently, the upsampled multi-scale low-level features are spliced ​​in the channel dimension to form a multi-scale low-level feature map containing information of different scales. Finally, a 1×1 convolution is used to compress the channels of the spliced ​​multi-scale low-level features, adjust the number of channels of the feature map, and send it to the DFAM module to enhance the ability to capture slender objects and obtain the shallow features of the spine image.

[0063] 3) Use the semantically rich spine features concatenated in S4 as input. These features contain high-frequency, low-frequency and anti-interference information and have strong semantic expression capabilities. Next, perform 1×1 convolution on the concatenated spine features, adjust the number of channels to match the input requirements of the DFAM module, and further fuse the feature information. Subsequently, the deep features after 1×1 convolution are input to the DFAM module to enhance the ability to capture slender objects. Finally, upsample the fused features output by the DFAM module to obtain the deep features of the spine image.

[0064] S6: Combine the main branch of S1 with the dynamic spectrum transformation branch of S3 to construct a segmentation network that fuses dynamic spectrum multi-branch features, and further fuse the shallow features and deep features obtained in S5 to obtain the spine segmentation result.

[0065] The multi-scale shallow features with different scale information obtained in step 5 are spliced ​​and fused with the deep features of the spine image enhanced by the DFAM module to capture slender objects, and a multi-scale, multi-semantic, adaptable fused spine feature with the ability to capture slender objects is obtained. Finally, after 3×3 convolution and upsampling operations, the output scale is adjusted and the final segmentation result is generated.

[0066] The present invention proposes a segmentation network with dynamic spectrum multi-branch feature fusion, which realizes multi-level optimization of book spine segmentation by combining with DFAM module. This method combines the main branch with the dynamic spectrum transformation branch, and fuses shallow and deep features, so that the segmentation network can not only flexibly adapt to multi-angle and slender objects in complex scenes, but also use DFAM's adaptive convolution and attention mechanism to enhance the ability to capture slender or irregular book spines. Through this multi-branch feature fusion method, the network has improved detail capture and anti-interference capabilities, so that it can handle complex bookshelf scenes such as irregular placement, making it superior to existing methods in segmentation accuracy and adaptability. This improvement effectively expands the application scenarios of the spine segmentation algorithm and provides stronger robustness for book segmentation in complex scenes.

[0067] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation, characterized in that: include: S1: The segmentation network framework adopts an encoding-decoding structure. The main branch of the encoding stage uses MobileNetV2 as the feature extraction network, combined with the dynamic spectrum transformation branch as the encoding part, to perform feature encoding and extract the semantic features of the spine image; S2: Send the semantic features of the spine image to the feature separation module to extract high-frequency features and low-frequency features respectively, and fuse the high-frequency features and low-frequency features to obtain high- and low-frequency fusion features; S3: Introduce a dynamic spectrum transformation branch, use the dynamic spectrum transformation module DST to perform frequency domain enhancement processing on the spine image, and obtain improved spine image features; S4: Combine the high-frequency features, low-frequency features, and high-low-frequency fusion features in S2 with the improved spine image features in S3 to obtain spine features with stronger adaptability and richer semantic information; S5: The DFAM module is introduced to extract the low-level features extracted after MobileNetV2 and the spine features obtained by S4, and obtain shallow and deep features capable of capturing slender objects; S6: Combine the main branch of S1 with the dynamic spectrum transformation branch of S3 to construct a segmentation network that fuses dynamic spectrum multi-branch features, and further fuse the shallow features and deep features obtained in S5 to obtain the spine segmentation result.

2. The spine segmentation method based on multi-branch feature fusion based on dynamic spectrum transformation according to claim 1 is characterized in that: The encoding-decoding structure described in S1 is composed of the following: the main branch MobileNetV2 feature extraction network and the dynamic spectrum transformation branch are the encoding part; the concatenation, 3×3 convolution and upsampling operations are the decoding part.

3. The spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation according to claim 1 is characterized in that: In S2, the features extracted by MobileNetV2 are sent to the feature separation module, and the features are divided into low-frequency features and high-frequency features through the pooling separation module. A self-attention mechanism is added to the high-frequency feature part to enhance the high-frequency feature information. At the same time, the enhanced high-frequency features are fused at multiple scales and then fused with the low-frequency features to obtain high-frequency and low-frequency fused features.

4. The spine segmentation method based on multi-branch feature fusion based on dynamic spectrum transformation according to claim 1 is characterized in that: In S3, the input book image is cut into pieces, and feature encoding is performed, and normalization and two-dimensional Fourier transform 2D FFT are performed in sequence to obtain the features of the spine image in the frequency domain; The characteristics of the spine image in the frequency domain are sent to the dynamic spectrum transformation module DST to reconstruct the low-frequency characteristics and introduce random interference to obtain adaptively enhanced spectrum characteristics; The adaptively enhanced spectral features are sequentially subjected to inverse Fourier transform (2DIFFT) and normalized to obtain improved spine image features.

5. The spine segmentation method based on multi-branch feature fusion of dynamic spectrum transformation according to claim 1 is characterized in that: In S5, the DFAM module is introduced into the segmentation network. First, in the main branch feature extraction stage, low-level features of different scales are upsampled and concatenated to obtain multi-scale low-level features. The obtained low-level features are sent to the DFAM module after 1×1 convolution to enhance the ability to capture slender objects and obtain shallow features of the spine image. The book spine features spliced ​​in S4 are subjected to 1×1 convolution and DFAM modules to enhance the ability to capture slender objects. The deep features of the book spine image are obtained through upsampling operations.

Images