A spine target detection method, system, device and storage medium
The spine target detection method, which combines an adaptive multipath resampling embedding module and a parallel scale fusion module, solves the problem of detection accuracy under complex conditions such as spine tilt and occlusion, and achieves high-precision and high-speed book inventory.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAMEN INTERACTIVE TECH
- Filing Date
- 2024-05-11
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies rely on complex post-processing corrections when dealing with complex situations such as spine tilt and occlusion, which affects segmentation accuracy and makes it difficult to meet the needs of high-speed book inventory.
A spine target detection method employing an adaptive multipath resampling embedding module and a parallel scale fusion module improves detection accuracy and robustness by adaptively adjusting the sampling path and fusing feature information at different scales. Combined with a lightweight network structure and optimization algorithm, the computational load is reduced.
It achieves high-precision spine target detection in complex backgrounds, meets the requirements of high-speed inventory, reduces computational load, and improves real-time performance.
Smart Images
Figure CN118447318B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, specifically to a method, system, device, and storage medium for detecting book spine targets. Background Technology
[0002] Book inventory is a crucial task in library management, reflecting the status of books on shelves, misplaced books, books checked out, and books lost. It forms the basis for services such as book retrieval, borrowing, and precise location. In the task of book inventory, solutions such as RFID tag-based book inventory and vision technology-based book inventory have emerged.
[0003] In her paper "Research on Intelligent Positioning Inventory Robots for Libraries—Taking Shanghai Library as an Example," Wu Jiayuan (Modern Computer Journal) summarized the technology and products of RFID-based book inventory robots. However, in the case of RFID inventory and OCR character recognition technology, the accuracy rate is 97% in ordinary scenarios and 87% in dense scenarios. But the initial book tagging and the construction and renovation of facilities for deploying RFID readers bring extremely high project implementation costs.
[0004] Based on computer vision technology, most literature and methods employ the two-stage horizontal object detection method MASK R-CNN (Mask Region-based Convolutional Neural Network) proposed by KM He, G Gkioxari et al., to achieve book spine segmentation. This is combined with OCR technology to extract spine content, and book inventory is then achieved through spine content retrieval. While this approach significantly reduces implementation costs, complex post-processing is required to correct for spindle tilting and occlusion, greatly impacting segmentation accuracy. Furthermore, the two-stage method struggles to meet the demands of high-speed inventory management.
[0005] In view of this, this application proposes a method, system, device and storage medium for detecting book spine targets, which can meet the needs of high-speed inventory and ensure high-precision detection results. Summary of the Invention
[0006] To address the problems in existing technologies that require complex post-processing for correction of issues such as spine tilt and occlusion, which significantly affects segmentation accuracy, this application provides a spine target detection method, system, device, and storage medium to solve the aforementioned technical deficiencies.
[0007] According to one aspect of this application, a method for detecting book spine targets is proposed, the method comprising the following steps:
[0008] S1. Obtain the target image of the spine of the book to be tested;
[0009] S2. Train the spine target detection model, which includes an adaptive multipath resampling embedding module and a parallel scale fusion module.
[0010] S3. Input the target image of the spine to be tested into the adaptive multipath resampling embedding module to extract and obtain the resampled image;
[0011] S4. Input the resampled image into the feature extraction module to extract the feature image, and input the feature image into the parallel scale fusion module to obtain the classification feature map, the target box feature map and the angle feature map respectively.
[0012] S5. The classification feature map, the target bounding box feature map, and the angle feature map are concatenated to obtain a fused feature map. The spine target detection model generates prediction results based on the fused feature map.
[0013] In the above technical solution, this application improves the accuracy and robustness of spine target detection by adaptively adjusting the sampling path and fusing feature information at different scales, especially for the detection effect of rotated and deformed spines.
[0014] Preferably, the adaptive multipath resampling embedding module in step S2 includes: a first convolutional base block, a high-resolution branch group, a low-resolution branch group, an upsampling fusion group, and a learnable super-resolution recovery fusion group.
[0015] More preferably, the first convolutional base block includes a convolutional kernel arranged in parallel and connected, a batch normalization layer, and an activation function layer;
[0016] High-resolution branch groups consist of two convolutional base blocks arranged in parallel and connected;
[0017] The low-resolution branch group consists of three convolutional base blocks arranged in parallel and connected;
[0018] The upsampling fusion group consists of a convolutional kernel, a batch normalization layer, and an upsampling layer;
[0019] The learnable super-resolution recovery fusion group includes a single pixel recombination operation and learnable gating coefficients.
[0020] Through the above technical solution, this application achieves efficient extraction and resampling of spine target features by constructing an adaptive multipath resampling embedding module, enabling the network to better adapt to spine targets of different scales, rotations and deformations.
[0021] Preferably, the parallel scale fusion module in step S2 includes: a high-dimensional channel fusion branch group, a mid-dimensional channel fusion branch group, a low-dimensional channel fusion branch group, and an attention group.
[0022] Further preferred, the high-dimensional channel fusion branch group includes the Deepy-SPP deep fusion module with spatial pyramid;
[0023] The mid-dimensional channel fusion branch group includes the Deepy deep fusion module and the S-Pre scale preprocessing module;
[0024] The low-dimensional channel fusion branch includes the Deepy deep fusion module and the S-Pre scale preprocessing module;
[0025] The Deepy deep fusion module includes a convolutional basic branch group and a residual module branch group, while the S-Pre scale preprocessing module includes an adaptive scale fusion residual unit and a convolutional downsampling unit.
[0026] In the above technical solution, this application uses parallel scale fusion technology to fuse feature information at different scales, thereby improving the network's robustness to complex backgrounds. Furthermore, this application employs a lightweight network structure and optimization algorithm, reducing computational load and improving real-time performance.
[0027] Preferably, in step S5, the spine target detection model generates a prediction result based on the fused feature map. The prediction result includes: prediction classification confidence, coordinates of the prediction box, and angle information; it also includes the following steps:
[0028] Calculate the intersection-union ratio (IUU) of the predicted bounding boxes and the true bounding boxes;
[0029] Based on the predicted classification confidence and intersection-union ratio, a matching matrix is constructed using the task alignment assignment method;
[0030] The top k samples are selected as positive examples based on the matching matrix, and the normalized matrix values are used as the weights of the loss function.
[0031] Further preferably, step S5 further includes: performing supervised learning on the classification task of the spine target detection model using the VFL loss function to obtain the VFL loss value, wherein the expression of the VFL loss function is:
[0032] VFL(pred,gt)=-1·weight·(gt·log(pred))+(1-gt)·log(1-pred))
[0033] Where, weight = 0.75·pred 2 ·(1-label)+gt·label
[0034] In the formula, gt represents the true value of the anchor point, pred represents the predicted value of the current anchor point, weight represents the weight of the binary cross-entropy of each batch, and label represents the category label in the current classification task.
[0035] The predicted and ground truth boxes are converted into Gaussian bounding boxes. The ProbIOU loss function is used to measure the similarity between the two distributions, and the ProbIOU loss value is obtained. The expression for the ProbIOU loss function is:
[0036]
[0037] In the formula, H D (p,q) represents the Hellinger distance between the distributions p and q, B C (p,q) are the Bach coefficients of the two distributions p and q;
[0038] The angle range is discretized, and the DFL loss function is used for supervised learning of the angle information prediction to obtain the DFL loss value. The expression of the DFL loss function is:
[0039] DFL=L left +L right =C(pred,w L ·[y]) left +C(pred,w R ·([y]+1)) right
[0040] In the formula, L left L represents the left loss function. right Let C denote the right loss function, C denote the cross-entropy loss function, y denote the label value, [y] denotes taking the integer part of y, and w L Indicates the left weight, w R Indicates right weight;
[0041] The VFL loss value, ProbIOU loss value, and DFL loss value are added together to obtain the total loss value. The weights of the spine target detection model are then updated using the backpropagation algorithm based on the total loss value.
[0042] In the above technical solution, this application can achieve high-performance target detection through carefully designed loss functions and inference optimization strategies during the training and detection phases.
[0043] Secondly, this application proposes a spine target detection system, which includes:
[0044] The image acquisition module is configured to acquire the target image of the spine of the book to be tested;
[0045] The model training module is configured to train the spine target detection model, which includes an adaptive multipath resampling embedding module and a parallel scale fusion module.
[0046] The resampled image acquisition module is configured to input the target image of the book spine to be tested into the adaptive multipath resampling embedding module to extract and obtain the resampled image;
[0047] The multi-feature map acquisition module is configured to input the resampled image into the feature extraction module to extract the feature image, and then input the feature image into the parallel scale fusion module to obtain the classification feature map, the target box feature map, and the angle feature map, respectively.
[0048] The prediction result generation module is configured to concatenate the classification feature map, the target bounding box feature map, and the angle feature map to obtain a fused feature map. The spine target detection model generates prediction results based on the fused feature map.
[0049] Thirdly, this application proposes a terminal device including a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the spine target detection method as described in any of the above.
[0050] Fourthly, this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the spine target detection method as described in any of the preceding claims.
[0051] Compared with the prior art, the beneficial results of the present invention are as follows:
[0052] (1) A high-precision rotating target detection technology based on computer vision and without anchor frames is adopted, which can meet the needs of high-speed inventory and ensure high-precision detection results. The prediction of the spine tilt angle is integrated into the prediction branch of the target detection network, and the tilt angle of the spine is learned end-to-end. The accurate spine target can be obtained without further post-processing correction.
[0053] (2) An adaptive multipath resampling embedding module is designed to achieve more efficient image dimensionality reduction. In this module, firstly, a multi-branch feature fusion is designed to enrich image features; secondly, a pixel recombination method is introduced to optimize blurred images and improve the ability to detect the spine of unclear images; finally, learnable gating coefficients are added, and the feature information of each branch is adaptively learned and fused through the backpropagation algorithm.
[0054] (3) Improved feature fusion network, namely parallel scale fusion module. For cases where there is a large difference in scale, such as between thin and thick spines in shelf images, the neck network module of feature fusion is improved to connect feature information from different scale branches, thereby enhancing scale perception. At the same time, an attention branch is added to adaptively learn an effective scale fusion method, further strengthening the model's ability to detect targets at different scales. Attached Figure Description
[0055] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments, taken with reference to the accompanying drawings:
[0056] Figure 1 This is a flowchart of the spine target detection method according to this application;
[0057] Figure 2 This is a framework diagram of the spine target detection method according to this application;
[0058] Figure 3a This is a schematic diagram of the convolutional module structure of the adaptive multipath resampling embedding module according to this application;
[0059] Figure 3b This is a schematic diagram of the feature map structure of the adaptive multipath resampling embedding module according to this application;
[0060] Figure 4 This is a schematic diagram of the structure of the parallel scale fusion module according to this application;
[0061] Figure 5 This is a structural diagram of the spine target detection system according to this application;
[0062] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation
[0063] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0064] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0065] Figure 1 A flowchart of the spine target detection method of this application is shown. Figure 2 A framework diagram of the spine target detection method of this application is shown, in conjunction with the reference. Figure 1 and Figure 2 The method for detecting book spine targets provided in this application includes the following steps:
[0066] S1. Obtain the target image of the spine of the book to be tested, and adjust the target image of the spine of the book to be tested to an RGB image with a size of 768×1024.
[0067] S2. Train the spine target detection model, which includes an adaptive multipath resampling embedding module and a parallel scale fusion module.
[0068] S3. Input the resized target image of the spine of the book to be tested into the adaptive multipath resampling embedding module to extract the resampled image.
[0069] Figure 3a The diagram shows a schematic of the convolutional module structure of the adaptive multipath resampling embedding module of this application. Please refer to [the diagram]. Figure 3a In this embodiment, the adaptive multipath resampling embedding module includes:
[0070] The system consists of a first convolutional base block of size 3×3, a high-resolution branch group, a low-resolution branch group, an upsampling fusion group, and a learnable super-resolution recovery fusion group.
[0071] The first convolutional base block, which is 3×3 in size, includes a convolutional kernel arranged horizontally in parallel and connected, a batch normalization layer, and an activation function layer.
[0072] High-resolution branch groups consist of two horizontally parallel and connected convolutional base blocks, sequentially using A... 1,1 A 1,2 express;
[0073] The low-resolution branch group consists of three horizontally parallel and connected convolutional base blocks, sequentially using A... 2,1 A 2,2 A 2,3 express;
[0074] The upsampling fusion group consists of a convolutional kernel, a batch normalization layer, and a nearest neighbor upsampling layer;
[0075] The learnable super-resolution restoration fusion group includes a pixel recombination operation and learnable gating coefficients, which are denoted by α.
[0076] Figure 3b This diagram illustrates the feature map structure of the adaptive multipath resampling embedding module of this application. Please refer to the reference. Figure 3a and Figure 3b In this embodiment, the first convolutional base block is used to pre-extract features from the target spine image and modify the feature channels of the target spine image. The output of the first convolutional base block is called the initial feature map, denoted by F. The high-resolution branch group and the low-resolution branch group are arranged in parallel, and the initial feature map F serves as the input to the high-resolution branch group and the low-resolution branch group.
[0077] The high-resolution branch group maintains the resolution of the initial feature map F, and the initial feature map F is input into A sequentially. 1,1 A1,2 Two convolutional base blocks yield a high-resolution feature map, which is then processed by F. 1,2 This indicates that high-resolution branch groups are used to extract shallow texture features.
[0078] High-resolution branch groups include A 1,1 A 1,2 Two convolutional basic blocks are connected in sequence, where A 1,1 The convolutional base block consists of: a 3×3 convolutional kernel, a batch of normalized layers, and a ReLU activation transform layer; the 3×3 convolutional kernel reduces the number of channels from 48 to 24 while maintaining the resolution of the initial feature map F. A 1,2 The basic convolutional block consists of: a 1×1 convolutional kernel, a batch of normalized layers, and a ReLU activation transform layer; the 1×1 convolutional kernel continues to maintain A 1,1 The output feature map resolution was adjusted, and the number of channels was restored from 24 to 48.
[0079] Low-resolution branch group, its A 2,1 The convolutional base block performs a 2x downsampling operation on the initial feature map F to obtain a low-resolution feature map, which is then used as F. 2,1 This indicates that the low-resolution feature map F 2,1 Enter A 2,2 The new low-resolution feature map obtained from the convolutional basic block is expressed using F. 2，2 Representation; Low-resolution feature map F 2，2 Enter A 2,3 The convolutional basic block yields a low-resolution feature map F. 2,3 The low-resolution branch group has a larger receptive field, and the low-resolution feature map F 2,3 It is a feature map with deeper features.
[0080] The low-resolution branch group includes A 2,1 A 2,2 A 2,3 The three convolutional basic blocks are connected in sequence, where A 2,1 The convolutional base block consists of: a 3×3 convolutional kernel, a batch of normalized layers, and a ReLU activation transform layer; the 3×3 convolutional kernel performs a 2x downsampling on the initial feature map F, while maintaining the number of channels at 48. A 2,2 The convolutional base block consists of: a 3×3 convolutional kernel, a batch of normalization layers, and a ReLU activation transformation layer; the 3×3 convolutional kernel increases the number of channels in the feature map from 48 to 64. A 2,3 The basic convolutional block consists of: a 1×1 convolutional kernel, a batch of normalization layers, and a ReLU activation transformation layer; the 1×1 convolutional kernel increases the number of channels in the feature map from 64 to 72.
[0081] The input to the upsampling fusion group is a low-resolution feature map F.2，2 The process involves sequentially performing a 1×1 convolutional layer, a batch normalization layer, and a 2x nearest neighbor upsampling operation to obtain an upsampled feature map. This is then analyzed using F... up Indicates; upsampled feature map F up Output feature map F of high-resolution branch group 1,2 Element-wise summation is performed to obtain a high-resolution feature map using F. 1,3 express.
[0082] The input to the learnable super-resolution recovery fusion group is a low-resolution feature map F. 2,3 First, execute a 1×1 convolutional layer to increase the channel dimension to C×2. 2 C is the low-resolution feature map F 2,3 The number of channels is then used to perform pixel recombination to generate a high-resolution feature map using F. 2,4 This indicates that the learnable gating coefficient α will interact with the high-resolution feature map F. 2,4 Multiplication, through backpropagation algorithm, allows α to adaptively adjust the high-resolution feature map F. 2,4 The fusion ratio can be learned, and the result obtained from the super-recovery fusion group is α×F. 2,4 .
[0083] Feature map F of high-resolution branch group 1,3 After performing a ReLU activation function transformation once, the transformed feature map F is obtained. R Transformation feature map F R The feature map α×F of the fusion group with learnable super-recovery 2,4 The features are added together to obtain the resampled feature map of the input feature extraction module, and then expressed as F. stem This means that the resampled image F stem =α×F 1,4 +F R .
[0084] Let the formula for the basic convolution block be: Where σ represents the activation layer and BN represents the batch normalization layer. Let f be the input feature map, in→out be the input and output channels, and i,j,k be the kernel sizes of the convolutional layers respectively. Then the formula for the adaptive multipath resampling embedding module is expressed as:
[0085]
[0086] In the formula, f is the initial feature map, and the square symbol represents the upsampling recovery operation.
[0087] Continue to refer to Figure 1 and Figure 2 The spine target detection method provided in this application further includes the following steps:
[0088] S4. Input the resampled image into the feature extraction module to extract the feature image. Input the feature image into the parallel scale fusion module to obtain the classification feature map f. cls Target bounding box feature map f reg and angular feature map f angle .
[0089] Figure 4 A schematic diagram of the parallel scale fusion module of this application is shown, as follows: Figure 4 As shown, in this embodiment, the neck network of the parallel scale fusion module includes:
[0090] The system consists of one high-dimensional channel fusion branch group, one mid-dimensional channel fusion branch group, one low-dimensional channel fusion branch group, and one attention group. Among them, the high-dimensional channel fusion branch group, the mid-dimensional channel fusion branch group, the low-dimensional channel fusion branch group, and the attention group are all in parallel.
[0091] In a specific embodiment, the feature extraction module includes feature extraction block 0, feature extraction block 1, feature extraction block 2, and feature extraction block 3 connected in sequence. The difference between each feature extraction block lies in the number of feature channels. The number of feature channels for feature extraction blocks 0-3 are 96, 192, 384, and 768, respectively.
[0092] With resampled image F STEM F0 is the input feature map for feature extraction block 0, and F1 is the output feature map for feature extraction block 1. F2 is the input feature map for feature extraction block 2, and F3 is the output feature map for feature extraction block 2. Feature map F3 is the feature map Γ. 1,1 Feature map F2 is feature map Γ” 2,1 The feature map F1 is the feature map Γ'3'. ,1 .
[0093] Feature extraction block i (i = 0, 1, 2, 3), taking i = 0 as an example, includes:
[0094] A 3×3 first convolutional base block, a residual module branch group, and a feature splicing attention module branch group.
[0095] The first convolutional base block is 3×3 in size, with a stride of 2. Feature map F STEM After the first convolutional base block, two identical feature maps are generated, which serve as the input feature maps for the residual module branch group and the feature splicing attention module branch group, respectively.
[0096] The residual module branch group consists of three residual modules. The first and third residual modules are residual unit structures of two 3×3 convolutional base blocks. The second residual module contains, in sequence, a 1×1 convolutional base block, a 3×3 convolutional base block, and a 1×1 convolutional base block. Let the output of the residual module branch group be...
[0097] The feature concatenation attention module branch group includes a 1×1 convolutional base block, and the feature map is... Assemble by channel and Recorded as And splicing feature maps The input channel attention module produces the output feature map F0.
[0098] The high-dimensional channel fusion branch executes the Deepy-SPP deep fusion module with spatial pyramid, which merges the feature map Γ. 1,1 The feature map Γ' is obtained by performing a 2x nearest neighbor upsampling. 2,1 ; Feature map Γ' 2,1 Input feature map Γ” fused with the mid-dimensional channel branch group 2,1 By concatenating the channels, we obtain the feature map Γ. 2,1 ; Feature map Γ 2,1 As the actual input to the mid-dimensional channel fusion branch, and after performing one Deepy deep fusion module and one S-Pre scale preprocessing module, the feature map Γ is obtained. 2,2 , Feature map Γ 2,2 The feature map Γ' is obtained by performing a 2x nearest neighbor upsampling. 3,1 Feature map Γ' 3,1 Input feature map Γ'3' fused with low-dimensional channels ,1 By concatenating the channels, we obtain the feature map Γ. 3,1 ; Feature map Γ 3,1 As the actual input to the low-dimensional channel fusion branch group, and after performing one Deepy deep fusion module and one S-Pre scale preprocessing module, the feature map Γ is obtained. 3,2 , Feature map Γ 3,2 The feature map Γ is obtained by performing 2x convolution downsampling. 4,1 Feature map Γ 4,1 As the actual input to the attention group, the ESE channel attention module, a 1×1 convolutional layer, a batch normalization layer, and a 2x nearest neighbor upsampling are executed sequentially to obtain the feature map.
[0099] The multi-scale fusion method of the neck network in the parallel scale fusion module: In the high-dimensional channel fusion branch group, the feature map Γ 1,1After performing a 1×1 convolutional layer and combining it with the feature map Element-wise addition, followed by a ReLU activation operation, yields the feature map Γ. 1,2 In the mid-dimensional channel fusion branch group, the feature map Γ 2,2 After performing a 1×1 convolutional layer and combining it with the feature map Element-wise addition, followed by a ReLU activation operation, yields the feature map Γ. 2,3 .
[0100] The output of each fusion group in the neck network of the parallel scale fusion module: the output feature map Γ in the low-dimensional channel fusion branch group. 3,2 It will execute a 3×3 convolutional kernel, a batch normalization layer, and then output a feature map with an attention group. Concatenate the data by channel, then perform a Leaky-ReLU activation layer to obtain the feature map Γ. 3,3 Feature map Γ 3,3 Executing a 1×1 convolutional kernel, a batch normalization layer, and a ReLU activation layer yields the output feature map Γ of the low-dimensional channel fusion branch group. 3,4 Feature map Γ 3,4 The feature map Γ is obtained by performing a 2x downsampling. 3,5 In the mid-dimensional channel fusion branch group, its characteristic map Γ 2,3 With feature map Γ 3,5 Feature map Γ is obtained by splicing according to channels. 24 Feature map Γ 24 Executing the Deepy module once yields the output feature map Γ of the mid-dimensional channel fusion branch group. 2,5 Feature map Γ 2,5 The feature map Γ is obtained by performing a 2x downsampling. 2,6 In the high-dimensional channel fusion branch group, its characteristic map Γ 1,2 With feature map Γ 2,6 Feature map Γ is obtained by splicing according to channels. 1,3 Feature map Γ 1,3 After executing the Deepy module, the output feature map Γ of the high-dimensional channel fusion branch group is obtained. 1,4 .
[0101] The Deepy deep fusion module includes: a single convolutional basic branch group and a residual module branch group.
[0102] The single convolutional basic branch group consists of a 3×3 convolutional basic block group, and the resulting feature map is denoted by f'. The residual module branch group consists of two residual units connected in sequence, and the resulting feature map is denoted by f”. Then, the feature map f' and the feature map f” are concatenated by channel to obtain the output feature map f of the Deepy module. The Deepy deep fusion module and the Deepy-SPP deep fusion module with spatial pyramid are structurally different only in that: in the residual module branch group, the SPP spatial pyramid module is embedded between the two residual units to obtain the Deepy-SPP deep fusion module with spatial pyramid. The SPP module copies the feature map three times, i.e., three branches, extracts features of different scales through different pooling sizes, and concatenates the feature maps output by the three branches by channel. Finally, a 3×3 convolutional basic block is executed to obtain the final feature map.
[0103] The S-Pre scale preprocessing module includes: an adaptive scale fusion residual unit and a convolutional downsampling unit.
[0104] The residual unit of adaptive scaling fusion includes: a first convolutional base block of size 1×1, a second convolutional base block of size 3×3, a third convolutional base block of size 1×1, a fourth convolutional base block of size 1×1, and a learnable gating coefficient.
[0105] The first, second, and third convolutional base blocks are arranged sequentially and connected, with the output denoted by f'. The number of input feature map channels remains unchanged. Simultaneously, the input feature map is input into the fourth convolutional base block, multiplied by the learnable gate coefficient, and the output is denoted by f”. Finally, the output of the residual unit for adaptive scale fusion is f' + f”. The convolutional downsampling unit, i.e., a 3×3 convolutional base block, reduces the resolution of the current feature map to its original size.
[0106] Continue to refer to Figure 1 The spine target detection method provided in this application further includes the following steps:
[0107] S5. Classification feature map f cls Target bounding box feature map f reg and angular feature map f angle The features are stitched together to obtain a fused feature map. The spine target detection model then generates prediction results based on this fused feature map. Specific prediction results include: prediction classification confidence, bounding box coordinates, and angle information.
[0108] In this embodiment, the following steps are also included:
[0109] Calculate the intersection-union ratio (IUR) between the predicted bounding boxes and the ground truth bounding boxes; construct a matching matrix using the task alignment assignment method based on the predicted classification confidence and IUR; select the top k positive samples based on the matching matrix, and use the normalized matrix values as the weights of the loss function.
[0110] The expression for the matching matrix is: t = s α +μ β Where s represents the prediction classification confidence, μ represents the intersection-union ratio of the predicted box and the ground truth box in the alignment learning of the rotation task, and α and β are hyperparameters, using the normalized matrix value (t value) as the weights of the loss function.
[0111] In a specific embodiment, the VFL loss function is used to perform supervised learning on the classification task of the spine object detection model to obtain the VFL loss value, wherein the expression of the VFL loss function is:
[0112] VFL(pred,gt)=-1·weight·(gt·log(pred))+(1-gt)·log(1-pred))
[0113] Where, weight = 0.75·pred 2 ·(1-label)+gt·label
[0114] In the formula, gt represents the true value of the anchor point, pred represents the predicted value of the current anchor point, weight represents the weight of the binary cross-entropy of each batch, and label represents the category label in the current classification task.
[0115] The predicted bounding boxes and ground truth boxes are transformed into Gaussian bounding boxes respectively. The similarity between the two distributions is measured by calculating their probability distributions. Supervised learning is then performed using the ProbIOU loss function, which is expressed as follows:
[0116]
[0117] In the formula, H D (p,q) represents the Hellinger distance between the distributions p and q, B C (p,q) are the Bach coefficients of the two distributions p and q.
[0118] For two Gaussian distributions p and q, let the mean and covariance matrix of distribution p be:
[0119]
[0120] The mean and covariance matrix of the q-distribution:
[0121]
[0122] Therefore, the Parseltongue distance B between p and q is... D It can be written as: B D =B1+B2, where
[0123]
[0124] therefore, Where ε is a hyperparameter used in practice to avoid calculation errors, and the value in this application is 0.001.
[0125] The angle range is discretized, specifically the angle [0, π / 2] is discretized into {x | x = 0, 1, 2, ..., 90}. Supervised learning is then performed on the prediction of angle information using the DFL loss function to obtain the DFL loss value. The expression for the DFL loss function is:
[0126] DFL=L left +L right =C(pred,w L ·[y]) left +C(pred,w R ·([y]+1)) right
[0127] In the formula, L left L represents the left loss function. right Let C denote the right loss function, C denote the cross-entropy loss function, y denote the label value, [y] denotes taking the integer part of y, and w L Indicates the left weight, w R The expressions for calculating the right weight, left weight, and right weight are as follows:
[0128] w L =[y]+1-y
[0129] w R =1-w L
[0130] The VFL loss value, ProbIOU loss value, and DFL loss value are added together to obtain the total loss value. The weights of the spine target detection model are then updated using the backpropagation algorithm based on the total loss value, thus realizing the spine target detection network.
[0131] In a specific embodiment, it further includes: using the angle integral formula Calculate the spine rotation angle. As mentioned earlier, angle prediction divides 0-90° into 91 intervals, therefore x represents the value of the current interval, and p... x The probability of falling within that interval is represented by ω = π / 180°. Then, the corner coordinates of the target spine are obtained through regression calculation, which constitutes the spine frame.
[0132] Further reference Figure 5 As an implementation of the above method, in a second aspect, this application provides an embodiment of a spine target detection system 500, which is similar to... Figure 1 Corresponding to the illustrated method embodiment, this system can be specifically applied to various electronic devices. The system 500 includes the following modules:
[0133] Image acquisition module 510 is configured to acquire the target image of the spine of the book to be tested;
[0134] The model training module 520 is configured to train the spine target detection model, which includes an adaptive multipath resampling embedding module and a parallel scale fusion module.
[0135] The resampled image acquisition module 530 is configured to input the target image of the spine to be tested into the adaptive multipath resampling embedding module to extract and obtain the resampled image.
[0136] The multi-feature map acquisition module 540 is configured to input the resampled image into the feature extraction module to extract the feature image, and input the feature image into the parallel scale fusion module to obtain the classification feature map, the target box feature map and the angle feature map respectively.
[0137] The prediction result generation module 550 is configured to concatenate the classification feature map, the target box feature map, and the angle feature map to obtain a fused feature map. The spine target detection model generates prediction results based on the fused feature map using a multi-class nonmaximum suppression method.
[0138] Thirdly, this application proposes a terminal device including a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the spine target detection method as described in any of the above.
[0139] Fourthly, this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the spine target detection method as described in any of the preceding claims.
[0140] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing terminal devices or servers in the embodiments of this application. Figure 6 The terminal device or server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0141] like Figure 6As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0142] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a liquid crystal display (LCD) and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card and a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.
[0143] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable medium or any combination thereof. The computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0144] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0146] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A method for detecting book spine targets, characterized in that, The method includes the following steps: S1. Obtain the image of the spine of the book to be tested; S2. Train a spine target detection model, wherein the spine target detection model includes an adaptive multipath resampling embedding module and a parallel scale fusion module; the adaptive multipath resampling embedding module includes: a first convolutional base block, a high-resolution branch group, a low-resolution branch group, an upsampling fusion group, and a learnable super-resolution recovery fusion group; the parallel scale fusion module includes: a high-dimensional channel fusion branch group, a mid-dimensional channel fusion branch group, a low-dimensional channel fusion branch group, and an attention group. S3. Input the target image of the spine to be tested into the adaptive multipath resampling embedding module to extract and obtain the resampled image; S4. Input the resampled image into the feature extraction module to extract the feature image, and input the feature image into the parallel scale fusion module to obtain the classification feature map, the target box feature map and the angle feature map respectively; S5. The classification feature map, the target box feature map, and the angle feature map are concatenated to obtain a fused feature map. The spine target detection model generates a prediction result based on the fused feature map.
2. The method for detecting book spine targets according to claim 1, characterized in that, The first convolutional base block includes a convolutional kernel arranged in parallel and connected, a batch normalization layer, and an activation function layer; The high-resolution branch group comprises two convolutional base blocks arranged in parallel and connected; The low-resolution branch group comprises three convolutional base blocks arranged in parallel and connected; The upsampling fusion group includes a convolutional kernel, a batch normalization layer, and an upsampling layer; The learnable super-resolution recovery fusion group includes a single pixel recombination operation and learnable gating coefficients.
3. The method for detecting book spine targets according to claim 1, characterized in that, The high-dimensional channel fusion branch group includes the Deepy-SPP deep fusion module with spatial pyramid; The mid-dimensional channel fusion branch group includes the Deepy deep fusion module and the S-Pre scale preprocessing module; The low-dimensional channel fusion branch group includes the Deepy deep fusion module and the S-Pre scale preprocessing module; The Deepy deep fusion module includes a convolutional basic branch group and a residual module branch group, and the S-Pre scale preprocessing module includes an adaptive scale fusion residual unit and a convolutional downsampling unit.
4. The method for detecting book spine targets according to claim 1, characterized in that, In step S5, the spine target detection model generates a prediction result based on the fused feature map. The prediction result includes: prediction classification confidence, coordinates of the prediction box, and angle information. The process also includes the following steps: Calculate the intersection-union ratio (IUU) of the predicted bounding box and the true bounding box; Based on the predicted classification confidence and the intersection-union ratio, a matching matrix is constructed using the task alignment assignment method; Positive samples are selected based on the matching matrix, and the normalized matrix values are used as weights for the loss function.
5. The spine target detection method according to claim 4, characterized in that, Step S5 further includes: performing supervised learning on the classification task of the spine target detection model using the VFL loss function to obtain the VFL loss value, wherein the expression of the VFL loss function is: in, In the formula, Indicates the truth value of the anchor point. This represents the current anchor point prediction value. The weights of the binary cross-entropy for each batch are represented. Indicates the category label in the current classification task; The predicted and ground truth boxes are converted into Gaussian bounding boxes, and the ProbIOU loss function is used to measure the similarity between the two distributions to obtain the ProbIOU loss value. The expression for the ProbIOU loss function is as follows: In the formula, express The Hellinger distance of the distribution yes The Bach coefficients of the two distributions; The angle range is discretized, and the prediction of the angle information is performed using the DFL loss function under supervised learning to obtain the DFL loss value. The expression of the DFL loss function is as follows: In the formula, Represents the left loss function. Represents the right loss function. Represents the cross-entropy loss function. Indicates the tag value. This means taking the integer part of y. Indicates left weight, Indicates right weight; The VFL loss value, ProbIOU loss value, and DFL loss value are added together to obtain the total loss value. The weights of the spine target detection model are then updated using the backpropagation algorithm based on the total loss value.
6. A spine target detection system, characterized in that, The system includes: The image acquisition module is configured to acquire the target image of the spine of the book to be tested; The model training module is configured to train a spine target detection model, wherein the spine target detection model includes an adaptive multipath resampling embedding module and a parallel scale fusion module; the adaptive multipath resampling embedding module includes: a first convolutional base block, a high-resolution branch group, a low-resolution branch group, an upsampling fusion group, and a learnable super-resolution recovery fusion group; the parallel scale fusion module includes: a high-dimensional channel fusion branch group, a mid-dimensional channel fusion branch group, a low-dimensional channel fusion branch group, and an attention group. The resampled image acquisition module is configured to input the target image of the spine to be tested into the adaptive multipath resampling embedding module to extract and obtain a resampled image. The multi-feature map acquisition module is configured to input the resampled image into the feature extraction module to extract feature images, and input the feature images into the parallel scale fusion module to obtain classification feature maps, target box feature maps, and angle feature maps, respectively. The prediction result generation module is configured to concatenate the classification feature map, the target box feature map, and the angle feature map to obtain a fused feature map, and the spine target detection model generates a prediction result based on the fused feature map.
7. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the spine target detection method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the spine target detection method as described in any one of claims 1 to 5.