Ancient book image detection method and system based on density map

Abstract

The present invention discloses an ancient book image detection method and system based on density map, which relates to the technical field of computer vision and deep learning. The method comprises: obtaining an original ancient book text image, inputting the original ancient book text image into an ancient book image detection network for detection, and outputting a detection result; wherein the ancient book image detection network is composed of an ancient text detection subnetwork and a semantic-based density map generation subnetwork, the ancient text detection subnetwork is used to detect the input original ancient book text image to obtain an initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and based on the density map, the original ancient book text image is cropped to obtain a predicted bounding box of a candidate area and the original image. The present invention improves the detection effect of small characters in ancient book images.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to an ancient book image detection method and system based on density maps. Background Art

[0002] Ancient books contain rich history, culture and profound philosophical thoughts. They are important material carriers of Chinese civilization and indispensable first-hand materials for studying ancient society, history and culture, especially the oracle bone inscriptions about 3,600 years ago. The digitization of ancient books can not only protect precious historical heritage, but also facilitate academic research and public access. In this context, ancient character detection has become a key step in automatic recognition and digital management. However, due to the long age and the influence of various natural disasters and human factors, ancient books have suffered varying degrees of damage, often accompanied by a large amount of severe and complex noise, which greatly reduces the integrity of ancient books and brings great obstacles to subsequent ancient character-related tasks. At the same time, they also face serious risks of damage and loss.

[0003] With the development of artificial intelligence technology, it provides strong technical support for the digitization of ancient books. In recent years, many scholars have conducted extensive research on the use of deep learning technology for ancient character detection. Researchers from Ritsumeikan University in Japan enhanced SSD to improve the detection accuracy of small characters in oracle bone inscriptions. In addition, researchers have constructed an OBI detection dataset and analyzed representative general detection frameworks, including Faster R-CNN, SSD, RefineDet, RFBnet, and Yolov3. The detection performance on this dataset has improved Yolov3 with the best performance. Recently, researchers have improved Yolov8 and added a small target detection head on its basis. Although previous work has achieved basic detection functions, there is still the problem of small characters being ignored due to uneven size, sparse distribution, and noise interference.

[0004] The current problems with ancient book image detection methods are as follows: (1) ancient characters are usually uneven in size and sparsely distributed; (2) the ancient characters in the image are mixed with a large number of small characters and the background noise is complex. These problems make the current ancient character detection methods ineffective in practical applications. Therefore, accurate detection of ancient characters remains a challenging task that requires robust and sophisticated methods. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an ancient book image detection method and system based on density map, which can effectively improve the detection accuracy of the ancient character detection model for small characters and has excellent generalization performance.

[0006] On the one hand, to achieve the above-mentioned purpose, the present invention provides an ancient book image detection method based on density map, comprising:

[0007] Acquire an original ancient book text image, input the original ancient book text image into an ancient book image detection network for detection, and output a detection result;

[0008] Among them, the ancient book image detection network is composed of an ancient character detection subnetwork and a semantic-based density map generation subnetwork. The ancient character detection subnetwork is used to detect the input original ancient book character image to obtain an initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and based on the density map, the original ancient book character image is cropped to obtain a predicted bounding box between the candidate area and the original image.

[0009] Preferably, the ancient character detection subnetwork adopts the Yolo series detector model as a detector and is trained on a public dataset.

[0010] Preferably, the semantic-based density map generation subnetwork includes a skeleton network layer and a visual attention mechanism layer; wherein the skeleton network layer is used to extract features of the image at different scales, and the visual self-attention mechanism layer is used to calculate the visual features of each region, and measure the similarity between the visual features of different regions and the semantic vectors, and determine the probability of the existence of text in each region.

[0011] Preferably, the skeleton network layer is a ResNet50 skeleton network layer, and the ResNet50 skeleton network layer extracts features of the second, third, and fourth modules, fuses features at different scales through a convolutional layer, and outputs a fused feature map.

[0012] Preferably, the visual self-attention mechanism layer calculates the attention score of each channel in the fused feature map using a channel attention mechanism, and calculates the attention score between the fused feature map and the semantic vector to obtain a semantic-based visual feature;

[0013] A similarity score between the semantic-based visual feature and the semantic vector is calculated, and the similarity score is adjusted through a channel attention mechanism to output a density map.

[0014] Preferably, the method of outputting the density map is:

[0015]

[0016] in, , are the learnable parameter matrices, is the semantic vector, To integrate visual features, is the sigmoid function, is the attention score for each channel, is a semantic-based visual feature. is the density map, is the convolution calculation, is the semantic-based visual feature on each channel, is the vector transpose operation, To traverse each channel, is a semantic vector.

[0017] Preferably, the original ancient book text image is cropped based on the density map, including:

[0018] A threshold is set based on the density map, an arbitrary area is selected in the density map, and the background or the area with low intensity in the selected area is filtered out to obtain a density mask map, wherein if the pixel intensity of the selected area is lower than the threshold, that is, the background or the area with low intensity, the pixel value of the selected area is modified to 0, and if the pixel intensity of the selected area is higher than the threshold, the pixel value of the selected area is modified to 1;

[0019] According to the density mask map, all connected pixels with a value of 1 are selected and merged to obtain a rectangular candidate area, and the original ancient book text image is cropped according to the rectangular candidate area to obtain a candidate detection result;

[0020] The candidate detection results and the original ancient text image are detected respectively to obtain the predicted bounding boxes of the candidate area and the original image.

[0021] Preferably, outputting the detection result includes:

[0022] The initial detection result, the candidate region and the predicted bounding box of the original image are fused and maximally suppressed to output the detection result.

[0023] On the other hand, to achieve the above-mentioned purpose, the present invention also provides an ancient book image detection system based on a density map, which is applied to an ancient book image detection method based on a density map, comprising:

[0024] Ancient book text image acquisition module: used to obtain the original ancient book text image;

[0025] Ancient book text detection module: used to detect the original ancient book text image through the ancient book image detection network and output the detection result, wherein the ancient book image detection network is composed of an ancient text detection subnetwork and a semantic-based density map generation subnetwork, the ancient text detection subnetwork is used to detect the input original ancient book text image to obtain the initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and based on the density map, the original ancient book text image is cropped to obtain the predicted bounding box of the candidate area and the original image.

[0026] Preferably, the ancient book text detection module includes a fusion unit, which is used to fuse the initial detection result, the candidate area and the predicted bounding box of the original image, and perform maximum suppression to output the detection result.

[0027] Compared with the prior art, the present invention has the following advantages and technical effects:

[0028] (1) This paper proposes an ancient book image detection network and applies density maps to ancient text detection. The density map is used to predict hot spots and integrate the detection results to construct an ancient book image detection method and system based on density maps, which solves the problems of poor small text detection in ancient text target detection and poor model generalization.

[0029] (2) The method proposed in the present invention improves the detection effect of small characters in ancient book images and can be applied to real scenes. It provides higher-quality technical support for the digitization of ancient books and the training of high-quality ancient character models, and has great utilization value. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 It is a flow chart of the ancient book image detection method based on density map according to an embodiment of the present invention;

[0032] Figure 2 A structural diagram of a sub-network generated from a semantic density graph according to an embodiment of the present invention;

[0033] Figure 3 It is a structural diagram of the ancient book image detection system based on density map according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0035] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The present invention proposes an ancient book image detection method based on density map, such as Figure 1 ,include:

[0037] Obtaining the original ancient book text image, inputting the original ancient book text image into the ancient book image detection network for detection, and outputting the detection result;

[0038] Among them, the ancient book image detection network is composed of an ancient character detection subnetwork and a semantic-based density map generation subnetwork. The ancient character detection subnetwork is used to detect the input original ancient book character image to obtain an initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and based on the density map, the original ancient book character image is cropped to obtain the predicted bounding box of the candidate area and the original image.

[0039] Furthermore, the ancient character detection sub-network adopts the Yolo series detector model as the detector and is trained on a public dataset.

[0040] Specifically, in this embodiment, the Yolov7 network is used and trained on a public data set of oracle bone inscription detection data set, with an input image of 512×512, to obtain an ancient inscription detection subnetwork.

[0041] Furthermore, the semantic-based density map generation subnetwork includes a skeleton network layer and a visual attention mechanism layer; wherein, the skeleton network layer is used to extract the features of the image at different scales, and the visual self-attention mechanism layer is used to calculate the visual features of each region, and measure the similarity between the visual features of different regions and the semantic vectors, and determine the probability of the existence of text in each region.

[0042] Specifically, density maps are widely used in object counting. Its advantage is that its integral can not only approximately represent the number of objects, but also the specific location and spatial distribution of objects. Considering that the text in the ancient book image is sparsely distributed, this embodiment uses a Gaussian kernel with a fixed kernel size and normalization to blur the center point of each text in the image to generate a true density map. Specifically, first obtain the image in the ancient text data set, and the coordinates of the center point of each text can be obtained according to the annotations. , and use the Gaussian kernel to blur the coordinates, and finally generate a density map with the same size as the original image. The sum of the pixel values ​​of the density map is the number of annotated text in the image. The formula is:

[0043]

[0044] in, n is the number of ancient characters in the image; i To go through every word; is the center point coordinate of the text; is the impulse function, when x = hour, =1, and 0 in other cases; is the Gaussian kernel, The density map generated.

[0045] The quality of the density map is extremely important for subsequent object detection based on the density map. Low-quality density maps will introduce more background noise and uneven text distribution. It may also cause incomplete text due to cropping, thereby misidentifying the radical as the entire text. Therefore, this embodiment uses a semantic-based density map generation subnetwork (DMNet) to improve the generation quality of the density map, such as Figure 2 shown.

[0046] In this embodiment, the method for evaluating the performance of the semantic-based density map generation sub-network is: using MAE (mean absolute error) and RMSE (root mean square error) as evaluation indicators to evaluate the performance of the semantic-based density map generation sub-network, as shown below:

[0047]

[0048] Where N represents the number of labeled objects in the image. represents the prediction result of the network, Represents the true value of the network.

[0049] Furthermore, the skeleton network layer in the semantic-based density map generation subnetwork is a ResNet50 skeleton network layer, and the ResNet50 skeleton network layer extracts features of the second, third, and fourth modules, fuses features at different scales through a convolutional layer, and outputs a fused feature map.

[0050] Specifically, the semantic-based density map generation subnetwork takes the original ancient book text image as input and outputs a density map that is downsampled 4 times the original image. The purpose is to predict the area where text exists in the original image from the contextual information provided by the density map, so as to correctly crop the image, maintain the integrity of the text, and enhance the detection ability of small objects.

[0051] The semantic-based density map generation subnetwork in this embodiment consists of two parts, namely a skeleton network layer based on ResNet50 and a semantic-based visual attention mechanism layer. The skeleton network layer is used to extract the features of the image at different scales, which is used to enhance the feature extraction capability and improve the generation quality of the density map. The purpose of the semantic-based self-attention mechanism layer is to calculate the visual features of each region and measure the similarity between it and the semantic vector, which is used to determine the probability of the existence of text in each region.

[0052] Furthermore, in this embodiment, the ResNet50 skeleton network layer is based on the pre-trained ResNet50, extracts the features of the second, third, and fourth modules of ResNet50, and fuses the features at each scale through a 1×1 convolutional layer, and outputs a fused visual feature that is 4 times the original image downsampled and has 256 channels.

[0053] Furthermore, the visual self-attention mechanism layer uses the channel attention mechanism to calculate the attention score of each channel in the fused feature map, and calculates the attention score between the fused feature map and the semantic vector to obtain semantic-based visual features;

[0054] The similarity score between the semantic-based visual features and the semantic vector is calculated, and the similarity score is adjusted through the channel attention mechanism to output a density map.

[0055] Specifically, the semantic-based visual attention mechanism layer requires two inputs, namely the fused feature map generated by the skeleton network layer and the semantic vector. First, the channel attention mechanism is used to calculate the attention score of each channel. Secondly, the attention score between the visual features and the semantic vector is calculated to obtain the semantic-based visual features . Then, the semantic-based visual features are calculated The similarity score between the word vector and the semantic vector is adjusted through the channel attention mechanism, and finally passes through a 1×1 convolution layer, with an output channel of 1 and a density map of the original image downsampled 4 times. Each pixel in the density map maps a 4×4 area in the original image, indicating the probability of text in the area.

[0056] Furthermore, the method to output the density map is:

[0057]

[0058] in, , are the learnable parameter matrices, is the semantic vector, To integrate visual features, is the sigmoid function, is the attention score for each channel, is a semantic-based visual feature. is the density map, is the convolution calculation, is the semantic-based visual feature on each channel, is the vector transpose operation, To traverse each channel, is the semantic vector;

[0059] Among them, the loss function used to train the semantic-based density map generation subnetwork is:

[0060]

[0061] in, are the parameters of the semantic-based density map generation module, is the total number of images in the training set, is the input image, The density map generated by the semantic-based density map generation subnetwork, is an image The real density map.

[0062] Furthermore, the original ancient book text image is cropped based on the density map, including:

[0063] A threshold is set based on the density map, an arbitrary area is selected in the density map, and the background or the area with low intensity in the selected area is filtered out to obtain a density mask map, wherein if the pixel intensity of the selected area is lower than the threshold, that is, the background or the area with low intensity, the pixel value of the selected area is modified to 0, and if the pixel intensity of the selected area is higher than the threshold, the pixel value of the selected area is modified to 1;

[0064] According to the density mask map, all connected pixels with a value of 1 are selected and merged to obtain a rectangular candidate area, and the original ancient book text image is cropped according to the rectangular candidate area to obtain a candidate detection result;

[0065] The candidate detection results and the original ancient text image are detected respectively to obtain the predicted bounding boxes of the candidate area and the original image.

[0066] Specifically, it can be seen from the generated density map that the pixel intensity in the area with more objects is higher than that in the area with fewer objects. Therefore, by setting a threshold in a region, it is possible to estimate whether there is an object in the region and filter out the background or low-intensity areas accordingly. If the pixel intensity is lower than the threshold, the pixel values ​​in the region are all 0, otherwise they are 1, and a density mask is generated. The formula is:

[0067]

[0068] in, is the indicator function, is the preset threshold, x is the pixel value.

[0069] In order to avoid the cropped image being too small, resulting in incomplete text in the area and causing false detection, a minimum cropping size is set. Since the density mask has no clear boundaries, the minimum rectangle algorithm is used to select all connected pixel areas with a mask value of 1 in the density mask and merge them into a large rectangular candidate area. The rectangular candidate area is then mapped to the original image and the original image is cropped.

[0070] Furthermore, the detection results are output, including:

[0071] The initial detection results, candidate regions and predicted bounding boxes of the original image are fused and greatly suppressed to output the detection results.

[0072] This embodiment also provides an ancient book image detection system based on density map, such as Figure 3 , applied to the ancient book image detection method based on density map, including:

[0073] Ancient book text image acquisition module: used to obtain the original ancient book text image;

[0074] Ancient book text detection module: used to detect the original ancient book text image through the ancient book image detection network and output the detection result, wherein the ancient book image detection network is composed of an ancient text detection subnetwork and a semantic-based density map generation subnetwork, the ancient text detection subnetwork is used to detect the input original ancient book text image to obtain the initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and based on the density map, the original ancient book text image is cropped to obtain the predicted bounding box of the candidate area and the original image.

[0075] Furthermore, the ancient book text detection module includes a fusion unit, which is used to fuse the initial detection result, the candidate area and the predicted bounding box of the original image to output the detection result.

[0076] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. The ancient book image detection method based on density map is characterized by: include: Acquire an original ancient book text image, input the original ancient book text image into an ancient book image detection network for detection, and output a detection result; The ancient book image detection network is composed of an ancient character detection subnetwork and a semantic-based density map generation subnetwork. The ancient character detection subnetwork is used to detect the input original ancient book character image to obtain an initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and based on the density map, the original ancient book character image is cropped to obtain a predicted bounding box between the candidate area and the original image; The semantic-based density map generation subnetwork includes a skeleton network layer and a visual attention mechanism layer; wherein the skeleton network layer is used to extract features of the image at different scales, and the visual attention mechanism layer is used to calculate the visual features of each region, and measure the similarity between the visual features of different regions and the semantic vectors, and determine the probability of the existence of text in each region; The skeleton network layer is a ResNet50 skeleton network layer, which extracts features of the second, third, and fourth modules, fuses features at different scales through a convolutional layer, and outputs a fused feature map; The visual attention mechanism layer calculates the attention score of each channel in the fused feature map by using the channel attention mechanism, and calculates the attention score between the fused feature map and the semantic vector to obtain a semantic-based visual feature; calculates the similarity score between the semantic-based visual feature and the semantic vector, and adjusts the similarity score by using the channel attention mechanism to output a density map; The method of outputting the density map is: ， in, , are the learnable parameter matrices, is the semantic vector, To integrate visual features, is the sigmoid function, is the attention score for each channel, is a semantic-based visual feature. is the density map, is the convolution calculation, is the semantic-based visual feature on each channel, is the vector transpose operation, To traverse each channel, is a semantic vector.

2. The ancient book image detection method based on density map according to claim 1 is characterized in that: The ancient character detection subnetwork adopts the Yolo series detector model as a detector and is trained on a public dataset.

3. The ancient book image detection method based on density map according to claim 1 is characterized in that: Cropping the original ancient book text image based on the density map includes: A threshold is set based on the density map, an arbitrary area is selected in the density map, and the background or the area with low intensity in the selected area is filtered out to obtain a density mask map, wherein if the pixel intensity of the selected area is lower than the threshold, that is, the background or the area with low intensity, the pixel value of the selected area is modified to 0, and if the pixel intensity of the selected area is higher than the threshold, the pixel value of the selected area is modified to 1; According to the density mask map, all connected pixels with a value of 1 are selected and merged to obtain a rectangular candidate area, and the original ancient book text image is cropped according to the rectangular candidate area to obtain a candidate detection result; The candidate detection results and the original ancient text image are detected respectively to obtain the predicted bounding boxes of the candidate area and the original image.

4. The ancient book image detection method based on density map according to claim 1 is characterized in that: Outputting the detection result includes: The initial detection result, the candidate region and the predicted bounding box of the original image are fused and maximally suppressed to output the detection result.

5. The ancient book image detection system based on density map is applied to the ancient book image detection method based on density map according to any one of claims 1 to 4, characterized in that: include: Ancient book text image acquisition module: used to obtain the original ancient book text image; Ancient book text detection module: used to detect the original ancient book text image through the ancient book image detection network and output the detection result, wherein the ancient book image detection network is composed of an ancient text detection subnetwork and a semantic-based density map generation subnetwork, the ancient text detection subnetwork is used to detect the input original ancient book text image to obtain an initial detection result; the semantic-based density map generation subnetwork is used to generate a density map, and the original ancient book text image is cropped based on the density map to obtain the predicted bounding box of the candidate area and the original image; The semantic-based density map generation subnetwork includes a skeleton network layer and a visual attention mechanism layer; the skeleton network layer is used to extract features of the image at different scales, and the visual attention mechanism layer is used to calculate the visual features of each region, and measure the similarity between the visual features of different regions and the semantic vectors, and determine the probability of the existence of text in each region; The skeleton network layer is a ResNet50 skeleton network layer, which extracts features of the second, third, and fourth modules, fuses features at different scales through a convolutional layer, and outputs a fused feature map; The visual attention mechanism layer calculates the attention score of each channel in the fused feature map by using the channel attention mechanism, and calculates the attention score between the fused feature map and the semantic vector to obtain a semantic-based visual feature; calculates the similarity score between the semantic-based visual feature and the semantic vector, and adjusts the similarity score by using the channel attention mechanism to output a density map; The method of outputting the density map is: ， in, , are the learnable parameter matrices, is the semantic vector, To integrate visual features, is the sigmoid function, is the attention score for each channel, is a semantic-based visual feature. is the density map, is the convolution calculation, is the semantic-based visual feature on each channel, is the vector transpose operation, To traverse each channel, is a semantic vector.

6. The ancient book image detection system based on density map according to claim 5 is characterized in that: The ancient book text detection module includes a fusion unit, which is used to fuse the initial detection result, the candidate area and the predicted bounding box of the original image, and perform maximum suppression to output the detection result.

Images