Method and device for extracting an image of a region of a potential artifact to be restored

Abstract

The application relates to the technical field of image extraction, in particular to a method and device for extracting a potential cultural relic to-be-repaired region image, which comprises the following steps: collecting a global image, and performing image preprocessing to obtain a preprocessed global image; establishing a to-be-repaired region image detection and recognition network, generating a boundary box for detecting a to-be-repaired region image based on the to-be-repaired region image detection and recognition network; cutting the preprocessed global image based on the boundary box; establishing a to-be-repaired region image segmentation network, segmenting the cut global image based on the to-be-repaired region image segmentation network, and obtaining a to-be-repaired region image; and performing precision evaluation on the to-be-repaired region image. Through the establishment of the to-be-repaired region image detection and recognition network and the segmentation network, the detection and recognition and segmentation of the to-be-repaired region image are realized, the extraction precision of the to-be-repaired region image can be greatly improved, and the efficiency of subsequent image repair work is improved.

Description

Technical Field

[0001] This invention relates to the field of image extraction technology, and in particular to a method and apparatus for extracting images of potential cultural relics to be restored. Background Technology

[0002] China, one of the ancient civilizations, possesses a wealth of cultural relics, a quintessential form of tangible cultural heritage, which plays a vital role in the research, inheritance, appreciation, and education of traditional culture. In contemporary China, cultural relics are an indispensable element in demonstrating cultural confidence. The rapid development of information and computer technology has made the digitization of cultural relics a crucial link in the entire process of digital archiving, digital restoration, and high-fidelity replication. With the emergence of artificial intelligence, utilizing advanced digitization and intelligent technologies to digitally display and effectively protect my country's ancient cultural heritage not only overcomes the difficulties and challenges of traditional methods but also solves problems that cannot be addressed by human effort in a faster and more efficient manner, possessing significant practical implications.

[0003] Currently, there are still many bottlenecks in the protection of large-format cultural relics, from theoretical methods to equipment development. Among them, the automatic location, identification, and extraction of potentially damaged areas of cultural relics is a major challenge. Most image restoration methods rely on manual determination of the areas to be restored before restoration begins. However, the areas to be restored are highly uncertain, and the numerous drawbacks of manually extracting damaged areas severely impact the efficiency and effectiveness of image restoration. Currently, some traditional image processing methods utilize information such as the edges, grayscale, and texture of damaged areas for automatic location. However, these methods lack adaptability and robustness to the imaging environment and type of cultural relics. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a method and apparatus for extracting images of potential cultural relics to be restored. By establishing an image detection and recognition network and a segmentation network for the area to be restored, the detection, recognition and segmentation of the image of the area to be restored can be realized, which can greatly improve the extraction accuracy of the image of the area to be restored and improve the efficiency of subsequent image restoration work.

[0005] This invention proposes a method for extracting images of potential cultural relic restoration areas, the method comprising:

[0006] Acquire a global image and perform image preprocessing on the global image to obtain a preprocessed global image;

[0007] A network for detecting and recognizing images of the region to be repaired is established, and bounding boxes for detecting images of the region to be repaired are generated based on the network.

[0008] The preprocessed global image is cropped based on the bounding box to obtain the cropped global image.

[0009] A segmentation network for the region to be repaired is established, and the cropped global image is segmented based on the segmentation network to obtain the image of the region to be repaired.

[0010] The accuracy of the image of the area to be repaired is evaluated.

[0011] Furthermore, the process of acquiring a global image and performing image preprocessing on the global image to obtain a preprocessed global image includes:

[0012] Perform geometric correction processing on the global image to obtain a geometrically corrected global image;

[0013] The global image after geometric correction is subjected to image fusion processing to obtain a preprocessed global image.

[0014] Furthermore, the step of establishing an image detection and recognition network for the region to be repaired, and generating bounding boxes for detecting the image of the region to be repaired based on the image detection and recognition network for the region to be repaired, includes:

[0015] A feature pyramid network is established, and feature extraction processing is performed on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales.

[0016] The feature extraction results at each scale are subjected to feature fusion processing to obtain multi-scale feature extraction results;

[0017] Based on the multi-scale feature extraction results, the image of the region to be repaired in the preprocessed global image is predicted to obtain several predicted bounding boxes.

[0018] Clustering and fusion processing is performed on the predicted bounding boxes to obtain bounding boxes used to detect the image of the region to be repaired.

[0019] Furthermore, the establishment of a feature pyramid network, and the feature extraction processing of the preprocessed global image based on the feature pyramid network, to obtain feature extraction results at various scales, include:

[0020] A three-layer feature pyramid network is established based on the Faster-RCNN feature extraction network. The preprocessed global image is input into the first-layer feature pyramid network to obtain the feature extraction prediction results of the first-layer feature pyramid network.

[0021] The global image processed by the first-layer feature pyramid network is input into the second-layer feature pyramid network to obtain the feature extraction and prediction results of the second-layer feature pyramid network.

[0022] The global image processed by the second-layer feature pyramid network is input into the third-layer feature pyramid network to obtain the feature extraction and prediction results of the third-layer feature pyramid network.

[0023] Furthermore, the step of establishing a feature pyramid network and performing feature extraction processing on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales also includes:

[0024] The third-layer feature pyramid network is copied to generate the fourth-layer feature pyramid network;

[0025] An upsampling operation is performed on the fourth-layer feature pyramid network to obtain the semantic feature information of the fourth-layer feature pyramid network;

[0026] The second-layer feature pyramid network is dimensionality reduced by using a convolutional layer with a kernel size of 1×1 to obtain high-resolution information of the second-layer feature pyramid network.

[0027] The semantic feature information of the fourth-layer feature pyramid network and the high-resolution information of the second-layer feature pyramid network are superimposed to generate the fifth-layer feature pyramid network.

[0028] An upsampling operation is performed on the fifth-layer feature pyramid network to obtain the semantic feature information of the fifth-layer feature pyramid network;

[0029] The first layer feature pyramid network is dimensionality reduced by using a convolutional layer with a kernel size of 1×1 to obtain high-resolution information of the first layer feature pyramid network.

[0030] The semantic feature information of the fifth-layer feature pyramid network and the high-resolution information of the first-layer feature pyramid network are superimposed to generate the sixth-layer feature pyramid network.

[0031] Furthermore, the feature fusion processing of the feature extraction results at each scale to obtain multi-scale feature extraction results includes:

[0032] The fourth, fifth, and sixth feature pyramid networks are convolutionally processed using convolutional layers with a kernel size of 3×3 to obtain multi-scale feature extraction results.

[0033] Furthermore, the clustering and fusion process performed on the plurality of predicted bounding boxes to obtain bounding boxes for detecting the image of the region to be repaired includes:

[0034] The predicted bounding boxes are clustered and fused using the K-means clustering algorithm to obtain the optimal aspect ratio for generating the anchor box.

[0035] Based on the optimal aspect ratio used to generate the anchor frame, a bounding box is generated for detecting the image of the region to be repaired.

[0036] Furthermore, the step of establishing an image segmentation network for the region to be repaired, and segmenting the cropped global image based on the image segmentation network to obtain the image of the region to be repaired, includes:

[0037] The cropped global image is processed using a ResNet101 deep residual network to obtain a feature map of the cropped global image.

[0038] The feature maps of the cropped global image are input into multi-layer pooling layers and multi-layer dilated convolutional layers respectively to extract feature information at different scales from the feature maps of the cropped global image;

[0039] The feature information at different scales in the feature map of the cropped global image is upsampled by a factor of 4, and then connected with the low feature map in the feature map of the cropped global image. After further upsampling, the final predicted feature map of the cropped global image is obtained. The final predicted feature map of the cropped global image has the same dimension as the cropped global image.

[0040] The cropped global image is segmented based on the final predicted feature map of the cropped global image to obtain the image of the region to be repaired.

[0041] Furthermore, the step of establishing an image segmentation network for the region to be repaired, and segmenting the cropped global image based on the image segmentation network to obtain the image of the region to be repaired, further includes:

[0042] Based on the multi-scale combined grouping method and the bounding box used to detect the image of the region to be repaired, the cropped global image is predicted to generate a rough image of the region to be repaired.

[0043] An FCN segmentation network framework is constructed, and the FCN segmentation network framework is trained based on a rough image of the region to be repaired, resulting in a trained FCN segmentation network framework.

[0044] Based on the trained FCN segmentation network framework, a coarse image of the region to be repaired is predicted to generate a second coarse image of the region to be repaired.

[0045] Repeat the above steps until a pixel-level detailed image of the area to be repaired is generated.

[0046] The present invention also proposes an apparatus for extracting images of potential cultural relic restoration areas, the apparatus comprising:

[0047] The acquisition and preprocessing module is used to acquire a global image and perform image preprocessing on the global image to obtain a preprocessed global image.

[0048] The detection and recognition module is used to establish a detection and recognition network for the image of the region to be repaired, and to generate bounding boxes for detecting the image of the region to be repaired based on the detection and recognition network.

[0049] An image cropping module is used to crop the preprocessed global image based on the bounding box to obtain a cropped global image.

[0050] An image segmentation module is used to establish an image segmentation network for the region to be repaired, and to segment the cropped global image based on the image segmentation network for the region to be repaired to obtain the image of the region to be repaired.

[0051] The accuracy assessment module is used to assess the accuracy of the image of the region to be repaired.

[0052] This invention establishes a detection and recognition network for the image of the region to be repaired, and uses a feature pyramid network for feature extraction. In addition to single-scale prediction via a bottom-up path, it also employs top-down and laterally connected paths. Utilizing strong semantic features at the top level and high-resolution information at the bottom level, it achieves feature classification and localization, significantly improving the performance of image feature extraction. Based on the K-means clustering algorithm, it generates the optimal aspect ratio of the bounding boxes, resulting in better detection performance for small targets and images of the region to be repaired with large aspect ratios. By establishing a segmentation network for the image of the region to be repaired, multi-scale fusion and refined structural design can significantly improve the segmentation effect and quality of small and weak images of the region to be repaired, achieving adaptive extraction of the image of the region to be repaired, greatly improving segmentation accuracy, and increasing the efficiency of subsequent image repair work. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the method for extracting images of potential cultural relics to be restored in an embodiment of the present invention;

[0055] Figure 2This is a flowchart of image preprocessing for a global image in an embodiment of the present invention;

[0056] Figure 3 This is a flowchart of generating bounding boxes for detecting the image of the region to be repaired in an embodiment of the present invention;

[0057] Figure 4 This is a flowchart illustrating the extraction of feature information at various scales from the preprocessed global image in an embodiment of the present invention.

[0058] Figure 5 This is a schematic diagram of the feature pyramid network structure in an embodiment of the present invention;

[0059] Figure 6 This is a flowchart illustrating the process of segmenting the cropped global image based on the image segmentation network of the region to be repaired, as described in this embodiment of the invention.

[0060] Figure 7 This is a schematic diagram of the device structure for extracting images of potential cultural relics to be restored in an embodiment of the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] In this invention, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, portions or combinations thereof disclosed in this specification, and are not intended to exclude the possibility that one or more other features, figures, steps, behaviors, components, portions or combinations thereof are present or added.

[0063] It should also be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0064] Example 1

[0065] This invention relates to a method for extracting images of potential cultural relic restoration areas. The method includes: acquiring a global image and performing image preprocessing on the global image to obtain a preprocessed global image; establishing a restoration area image detection and recognition network, and generating bounding boxes for detecting the restoration area image based on the restoration area image detection and recognition network; cropping the preprocessed global image based on the bounding boxes to obtain a cropped global image; establishing a restoration area image segmentation network, and segmenting the cropped global image based on the restoration area image segmentation network to obtain the restoration area image; and evaluating the accuracy of the restoration area image.

[0066] In one optional implementation of this embodiment, such as Figure 1 As shown, Figure 1 The flowchart of a method for extracting images of potential cultural relic restoration areas according to an embodiment of the present invention is shown, including the following steps:

[0067] S101. Acquire a global image and perform image preprocessing on the global image to obtain a preprocessed global image;

[0068] In an optional implementation of this embodiment, the global image is acquired via the Internet or via an acquisition device.

[0069] In one optional implementation of this embodiment, such as Figure 2 As shown, Figure 2 The following flowchart illustrates an embodiment of the present invention for image preprocessing of a global image, including the following steps:

[0070] S201. Perform geometric correction processing on the global image to obtain a geometrically corrected global image;

[0071] In an optional implementation of this embodiment, the geometric correction process refers to eliminating geometric deformations in the global image, eliminating internal distortions in the global image, and achieving geometric integration between the global image and the standard image.

[0072] S202. Perform image fusion processing on the global image after geometric correction to obtain a preprocessed global image.

[0073] In an optional implementation of this embodiment, the image fusion processing refers to extracting the beneficial information from the multi-source channels of the global image after geometric correction processing to the greatest extent possible, and synthesizing a high-quality global image.

[0074] S102. Establish an image detection and recognition network for the region to be repaired, and generate bounding boxes for detecting the image of the region to be repaired based on the image detection and recognition network for the region to be repaired.

[0075] In one optional implementation of this embodiment, such as Figure 3 As shown, Figure 3 The flowchart illustrating the generation of bounding boxes for detecting the region to be repaired in an embodiment of the present invention includes the following steps:

[0076] S301. Establish a feature pyramid network, and perform feature extraction processing on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales.

[0077] In one optional implementation of this embodiment, the Feature Pyramid Network (FPN) is a neural network architecture for object detection. Traditional object detection algorithms often require multiple processing cycles of the same image to detect objects at different scales. This is because the feature maps used in traditional algorithms are obtained from the last convolutional layer, thus they can only detect smaller objects and are ineffective for larger ones. However, the Feature Pyramid Network extracts feature information at different scales from different feature layers, outputting feature maps of multiple scales. These feature maps are then fused using top-down and bottom-up paths, ultimately generating a feature pyramid with multi-scale information.

[0078] Specifically, such as Figure 4 and Figure 5 As shown, Figure 4 This invention illustrates a flowchart of extracting feature information at various scales from a preprocessed global image in an embodiment of the invention. Figure 5 A schematic diagram of the feature pyramid network structure in an embodiment of the present invention is shown, including the following steps:

[0079] S401. A three-layer feature pyramid network is established based on the Faster-RCNN feature extraction network. The preprocessed global image is input into the first-layer feature pyramid network to obtain the feature extraction prediction result of the first-layer feature pyramid network.

[0080] Specifically, Faster-RCNN is a convolutional network model for object detection that integrates feature extraction, target region lookup, and bounding box regression into a single network, achieving end-to-end learning and significantly improving detection capability and speed. Figure 5 As shown, in this embodiment of the invention, a three-layer feature pyramid network is established based on the Faster-RCNN feature extraction network.

[0081] In an optional implementation of this embodiment, the preprocessed global image is input into the first-layer feature pyramid network, and the feature extraction prediction result of the first-layer feature pyramid network is obtained after prediction by the first-layer feature pyramid network.

[0082] S402. Input the global image processed by the first-layer feature pyramid network into the second-layer feature pyramid network to obtain the feature extraction and prediction results of the second-layer feature pyramid network.

[0083] In an optional implementation of this embodiment, the global image processed by the first-layer feature pyramid network is input into the second-layer feature pyramid network, and the feature extraction prediction result of the second-layer feature pyramid network is obtained after prediction processing by the second-layer feature pyramid network.

[0084] S403. Input the global image processed by the second-layer feature pyramid network into the third-layer feature pyramid network to obtain the feature extraction and prediction results of the third-layer feature pyramid network.

[0085] In an optional implementation of this embodiment, the global image processed by the second-layer feature pyramid network is input into the third-layer feature pyramid network, and the feature extraction and prediction result of the third-layer feature pyramid network is obtained after prediction processing by the third-layer feature pyramid network.

[0086] It should be noted that the scale ratio of adjacent feature pyramid networks is 2, that is, the stride of the first feature pyramid network relative to the preprocessed global image is 2, the stride of the second feature pyramid network relative to the preprocessed global image is 4, and the stride of the third feature pyramid network relative to the preprocessed global image is 8.

[0087] S404. Copy the third-layer feature pyramid network to generate a fourth-layer feature pyramid network;

[0088] S405. Perform an upsampling operation on the fourth-layer feature pyramid network to obtain the semantic feature information of the fourth-layer feature pyramid network. Perform dimensionality reduction processing on the second-layer feature pyramid network based on a convolutional layer with a kernel size of 1×1 to obtain the high-resolution information of the second-layer feature pyramid network. Superimpose the semantic feature information of the fourth-layer feature pyramid network and the high-resolution information of the second-layer feature pyramid network to generate the fifth-layer feature pyramid network.

[0089] In one optional implementation of this embodiment, a typical feature pyramid network extracts different features at each layer and then performs corresponding predictions. Alternatively, it obtains feature maps of different sizes through convolution and pooling operations and uses the feature map from the last convolutional layer for prediction. Or, it performs depthwise convolution on the original image and performs predictions at different feature layers. All three methods have certain drawbacks, including high computational cost, focusing only on the features of the last layer while ignoring features from other layers, and a lack of adaptability and robustness in the obtained features. Therefore, in this embodiment of the invention, a combination of top-down, bottom-up, convolutional fusion, and lateral connections is used, while leveraging the strong semantic features of higher-level features and combining them with the high resolution of lower-level features to obtain more accurate features.

[0090] In an optional implementation of this embodiment, a 2x upsampling operation is performed on the fourth-layer feature pyramid network, that is, the fourth-layer feature pyramid network is enlarged to the same size as the feature map in the second-layer feature pyramid network.

[0091] In an optional implementation of this embodiment, the second-layer feature pyramid network is dimensionality-reduced based on a convolutional layer with a kernel size of 1×1 to obtain high-resolution information of the second-layer feature pyramid network. Here, the number of channels in the second-layer feature pyramid network is corrected to be consistent with that of the fourth-layer feature pyramid network.

[0092] Furthermore, the semantic feature information of the fourth-layer feature pyramid network, which is beneficial for classification, and the high-resolution information of the second-layer feature pyramid network, which is beneficial for localization, are superimposed here to generate the fifth-layer feature pyramid network.

[0093] S406. Perform an upsampling operation on the fifth-layer feature pyramid network to obtain the semantic feature information of the fifth-layer feature pyramid network. Perform dimensionality reduction processing on the first-layer feature pyramid network based on a convolutional layer with a kernel size of 1×1 to obtain the high-resolution information of the first-layer feature pyramid network. Superimpose the semantic feature information of the fifth-layer feature pyramid network and the high-resolution information of the first-layer feature pyramid network to generate the sixth-layer feature pyramid network.

[0094] It should be noted that the implementation principle of step S406 is basically the same as that of step S405, and will not be described in detail here.

[0095] S302. Perform feature fusion processing on the feature extraction results of each scale to obtain multi-scale feature extraction results;

[0096] In an optional implementation of this embodiment, the fourth, fifth, and sixth feature pyramid networks are convolved based on convolutional layers with a kernel size of 3×3 to obtain multi-scale feature extraction results.

[0097] Specifically, based on convolutional layers with a kernel size of 3×3, the fourth, fifth, and sixth layers of the feature pyramid network are analogous to the features of each layer in the feature pyramid network. Regions of interest (ROIs) of different scales are mapped to the corresponding feature layers. In this embodiment, taking a 224-pixel image input as an example, the ROI with width w and height h will be mapped to feature level k, calculated as follows:

[0098]

[0099] S303. Based on the multi-scale feature extraction results, predict the image of the region to be repaired in the preprocessed global image to obtain several predicted bounding boxes.

[0100] In one optional implementation of this embodiment, the image of the region to be repaired in the preprocessed global image is predicted based on the obtained multi-scale feature extraction results to obtain several predicted bounding boxes, the sizes of which differ to varying degrees.

[0101] S304. Perform clustering and fusion processing on the predicted bounding boxes to obtain bounding boxes used to detect the image of the region to be repaired.

[0102] In an optional implementation of this embodiment, the predicted bounding boxes are clustered and fused based on the K-means clustering algorithm to obtain the optimal aspect ratio for generating anchor boxes, and bounding boxes for detecting the image of the region to be repaired are generated based on the optimal aspect ratio for generating anchor boxes.

[0103] It should be noted that the image detection and recognition network for the region to be repaired is quite sensitive to changes in the scale and proportion of the object's bounding box. In this embodiment of the invention, the bounding boxes are first clustered and fused to generate the optimal aspect ratio for generating the anchor box. Since this optimal aspect ratio embeds prior information about the shape of the target to be detected, it will have a better detection effect on small targets and images of the region to be repaired with a large aspect ratio.

[0104] In an optional implementation of this embodiment, the K-means clustering algorithm, also known as the k-means clustering algorithm, is an iterative clustering analysis algorithm. In this embodiment, several bounding boxes are divided into K groups, K objects are randomly selected as initial cluster centers, and the distance between each bounding box and the bounding box that serves as the cluster center is calculated. Each bounding box is assigned to the nearest cluster center. Each time a sample bounding box is assigned, the distance is recalculated based on the existing bounding box objects until no bounding box object can be reassigned to a different cluster center. Using this algorithm, the sum of squared errors between bounding boxes can be minimized locally, thereby obtaining the optimal aspect ratio for generating anchor bounding boxes.

[0105] S103. Based on the bounding box, crop the preprocessed global image to obtain the cropped global image;

[0106] In an optional implementation of this embodiment, the preprocessed global image is cropped based on the bounding box with the optimal aspect ratio obtained in step S102 to obtain a cropped global image, which facilitates the subsequent extraction of the image of the region to be repaired from the cropped global image.

[0107] S104. Establish a segmentation network for the region to be repaired, and segment the cropped global image based on the segmentation network to obtain the image of the region to be repaired.

[0108] In one optional implementation of this embodiment, such as Figure 6 As shown, Figure 6 The following flowchart illustrates a process for segmenting the cropped global image using an image segmentation network based on the region to be repaired, as described in an embodiment of the present invention. The process includes the following steps:

[0109] S601. Based on the ResNet101 deep residual network, feature processing is performed on the cropped global image to obtain the feature map of the cropped global image.

[0110] In an optional implementation of this embodiment, a ResNet101 deep residual network is used, with dilated convolutional layers added to the last three convolutional layers. The cropped global image is then input into the ResNet101 deep residual network for feature processing to obtain the feature map of the cropped global image.

[0111] S602. Input the feature map of the cropped global image into a multi-layer pooling layer and a multi-layer dilated convolutional layer respectively, and extract feature information of different scales from the feature map of the cropped global image.

[0112] In one optional implementation of this embodiment, an image segmentation network for the region to be repaired is built based on an encoder-decoder architecture. Several pooling layers and dilated convolutional layers of different sizes are designed in the image segmentation network for the region to be repaired to extract feature information of different scales from the feature map of the cropped global image.

[0113] S603. Upsample the feature information of different scales in the feature map of the cropped global image by a factor of 4, connect it with the low feature map in the feature map of the cropped global image, and perform upsampling operation to obtain the final predicted feature map of the cropped global image.

[0114] In an optional implementation of this embodiment, feature information at different scales is converted to the same scale as the low feature map in the feature map of the cropped global image, and upsampled to the same dimension. The final predicted feature map of the cropped global image has the same dimension as the cropped global image.

[0115] S604. Based on the final predicted feature map of the cropped global image, the cropped global image is segmented to obtain the image of the region to be repaired.

[0116] By employing multi-scale fusion and refined structural design, the segmentation effect and quality of small and weak regions to be repaired can be significantly improved. This enables adaptive extraction of the images of the regions to be repaired, greatly enhancing segmentation accuracy and increasing efficiency for subsequent image repair work.

[0117] In one optional implementation of this embodiment, a weakly supervised learning method based on bounding box annotation can be used to generate a pixel-level fine image of the region to be repaired based on a multi-scale combination grouping method and the bounding boxes used to detect the image of the region to be repaired.

[0118] Specifically, an FCN (Fully Convolutional Networks for Semantic Segmentation) segmentation network framework is constructed, and a multi-scale combination grouping method MCG (Master Cell Group) is used to train the FCN segmentation network framework based on a coarse image of the region to be repaired. The semantic segmentation capability of the segmentation network framework is improved through continuous iteration until a pixel-level fine image of the region to be repaired is generated.

[0119] S105. The accuracy of the image of the area to be repaired is evaluated.

[0120] In one optional implementation of this embodiment, the accuracy of the image to be repaired is evaluated by calculating the overall precision, recall, precision, crossover ratio, mean crossover ratio, and kappa coefficient of the image to be repaired.

[0121] In summary, Embodiment 1 of this invention proposes a method for extracting images of potential cultural relic restoration areas. By establishing an image detection and recognition network for the restoration area and utilizing a feature pyramid network for feature extraction, in addition to single-scale prediction via a bottom-up path, a top-down and horizontally connected path is employed. Leveraging the strong semantic features at the top layer and the high-resolution information at the bottom layer, feature classification and localization are achieved, significantly improving the performance of image feature extraction. The optimal aspect ratio of the bounding boxes is generated based on the K-means clustering algorithm, resulting in better detection performance for small targets and images of restoration areas with large aspect ratios. Furthermore, by establishing an image segmentation network for the restoration area, multi-scale fusion and refined structural design can significantly improve the segmentation effect and quality of small and weak restoration area images, enabling adaptive extraction of restoration area images and greatly improving segmentation accuracy, thus increasing efficiency for subsequent image restoration work.

[0122] Example 2

[0123] This invention also relates to a device for extracting images of potential cultural relic restoration areas, such as... Figure 7 As shown, Figure 7 This diagram illustrates a device for extracting images of potential cultural relic restoration areas according to an embodiment of the present invention. The device includes:

[0124] The acquisition and preprocessing module 10 is used to acquire a global image and perform image preprocessing on the global image to obtain a preprocessed global image.

[0125] The detection and recognition module 20 is used to establish a detection and recognition network for the image of the region to be repaired, and to generate bounding boxes for detecting the image of the region to be repaired based on the detection and recognition network.

[0126] Image cropping module 30, the image cropping module is used to crop the preprocessed global image based on the bounding box to obtain the cropped global image;

[0127] Image segmentation module 40, the image segmentation module is used to establish an image segmentation network for the region to be repaired, and to segment the cropped global image based on the image segmentation network for the region to be repaired to obtain the image of the region to be repaired;

[0128] The accuracy assessment module 50 is used to assess the accuracy of the image of the area to be repaired.

[0129] In summary, Embodiment 2 of this invention proposes an apparatus for extracting images of potential cultural relic restoration areas. This apparatus executes a method for extracting images of potential cultural relic restoration areas. By establishing an image detection and recognition network for the restoration area and utilizing a feature pyramid network for feature extraction, it employs both bottom-up path prediction at a single scale and top-down and horizontally connected paths. Utilizing strong semantic features at the top layer and high-resolution information at the bottom layer, it achieves feature classification and localization, significantly improving the performance of image feature extraction. Based on the K-means clustering algorithm, it generates the optimal aspect ratio of bounding boxes, resulting in better detection performance for small targets and images of restoration areas with large aspect ratios. By establishing an image segmentation network for the restoration area, multi-scale fusion and refined structural design significantly improve the segmentation effect and quality of small and weak restoration area images, enabling adaptive extraction of restoration area images and greatly improving segmentation accuracy, thus increasing efficiency for subsequent image restoration work.

[0130] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0131] Furthermore, the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for extracting images of potential cultural relic restoration areas, characterized in that, The method includes: Acquire a global image and perform image preprocessing on the global image to obtain a preprocessed global image; A network for detecting and recognizing images of the region to be repaired is established, and bounding boxes for detecting images of the region to be repaired are generated based on the network. The step of establishing a network for detecting and recognizing the image of the region to be repaired, and generating bounding boxes for detecting the image of the region to be repaired based on the network, includes: establishing a feature pyramid network; performing feature extraction processing on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales; performing feature fusion processing on the feature extraction results at various scales to obtain multi-scale feature extraction results; predicting the image of the region to be repaired in the preprocessed global image based on the multi-scale feature extraction results to obtain several predicted bounding boxes; and performing clustering and fusion processing on the several predicted bounding boxes to obtain bounding boxes for detecting the image of the region to be repaired. The process of establishing a feature pyramid network and performing feature extraction on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales includes: establishing a three-layer feature pyramid network based on the Faster-RCNN feature extraction network; inputting the preprocessed global image into the first-layer feature pyramid network to obtain the feature extraction prediction results of the first-layer feature pyramid network; inputting the global image processed by the first-layer feature pyramid network into the second-layer feature pyramid network to obtain the feature extraction prediction results of the second-layer feature pyramid network; and inputting the global image processed by the second-layer feature pyramid network into the third-layer feature pyramid network to obtain the feature extraction prediction results of the third-layer feature pyramid network. The process of establishing a feature pyramid network and extracting features from the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales further includes: copying the third-layer feature pyramid network to generate a fourth-layer feature pyramid network; performing an upsampling operation on the fourth-layer feature pyramid network to obtain its semantic feature information; performing dimensionality reduction processing on the second-layer feature pyramid network based on a convolutional layer with a kernel size of 1×1 to obtain its high-resolution information; superimposing the semantic feature information of the fourth-layer feature pyramid network and the high-resolution information of the second-layer feature pyramid network to generate a fifth-layer feature pyramid network; performing an upsampling operation on the fifth-layer feature pyramid network to obtain its semantic feature information; performing dimensionality reduction processing on the first-layer feature pyramid network based on a convolutional layer with a kernel size of 1×1 to obtain its high-resolution information; and superimposing the semantic feature information of the fifth-layer feature pyramid network and the high-resolution information of the first-layer feature pyramid network to generate a sixth-layer feature pyramid network. The preprocessed global image is cropped based on the bounding box to obtain the cropped global image. A segmentation network for the region to be repaired is established, and the cropped global image is segmented based on the segmentation network to obtain the image of the region to be repaired. The process of establishing a segmentation network for the region to be repaired, and segmenting the cropped global image based on this network to obtain the image of the region to be repaired, includes: performing prediction processing on the cropped global image based on a multi-scale grouping method and the bounding boxes used to detect the region to be repaired, generating a coarse image of the region to be repaired; constructing an FCN segmentation network framework, training the FCN segmentation network framework based on the coarse image of the region to be repaired, obtaining a trained FCN segmentation network framework; performing prediction processing on the coarse image of the region to be repaired based on the trained FCN segmentation network framework, generating a sub-coarse image of the region to be repaired; repeating the above steps until a pixel-level fine image of the region to be repaired is generated. The accuracy of the image of the area to be repaired is evaluated.

2. The method for extracting images of potential cultural relic restoration areas as described in claim 1, characterized in that, The process of acquiring a global image and performing image preprocessing on the global image to obtain a preprocessed global image includes: Perform geometric correction processing on the global image to obtain a geometrically corrected global image; The global image after geometric correction is subjected to image fusion processing to obtain a preprocessed global image.

3. The method for extracting images of potential cultural relic restoration areas as described in claim 1, characterized in that, The feature fusion processing of the feature extraction results at each scale to obtain multi-scale feature extraction results includes: The fourth, fifth, and sixth feature pyramid networks are convolutionally processed using convolutional layers with a kernel size of 3×3 to obtain multi-scale feature extraction results.

4. The method for extracting images of potential cultural relic restoration areas as described in claim 1, characterized in that, The step of clustering and fusion of the predicted bounding boxes to obtain bounding boxes for detecting the region to be repaired in the image includes: The predicted bounding boxes are clustered and fused using the K-means clustering algorithm to obtain the optimal aspect ratio for generating the anchor box. Based on the optimal aspect ratio used to generate the anchor frame, a bounding box is generated for detecting the image of the region to be repaired.

5. The method for extracting images of potential cultural relic restoration areas as described in claim 1, characterized in that, The step of establishing an image segmentation network for the region to be repaired, and segmenting the cropped global image based on the image segmentation network to obtain the image of the region to be repaired, includes: The cropped global image is processed using a ResNet101 deep residual network to obtain a feature map of the cropped global image. The feature maps of the cropped global image are input into multi-layer pooling layers and multi-layer dilated convolutional layers respectively to extract feature information at different scales from the feature maps of the cropped global image; The feature information at different scales in the feature map of the cropped global image is upsampled by a factor of 4, and then connected with the low feature map in the feature map of the cropped global image. After further upsampling, the final predicted feature map of the cropped global image is obtained. The final predicted feature map of the cropped global image has the same dimension as the cropped global image. The cropped global image is segmented based on the final predicted feature map of the cropped global image to obtain the image of the region to be repaired.

6. An apparatus for extracting images of potential cultural relic restoration areas, characterized in that, The device includes: The acquisition and preprocessing module is used to acquire a global image and perform image preprocessing on the global image to obtain a preprocessed global image. The detection and recognition module is used to establish a detection and recognition network for the image of the region to be repaired, and to generate bounding boxes for detecting the image of the region to be repaired based on the detection and recognition network. The step of establishing a network for detecting and recognizing the image of the region to be repaired, and generating bounding boxes for detecting the image of the region to be repaired based on the network, includes: establishing a feature pyramid network; performing feature extraction processing on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales; performing feature fusion processing on the feature extraction results at various scales to obtain multi-scale feature extraction results; predicting the image of the region to be repaired in the preprocessed global image based on the multi-scale feature extraction results to obtain several predicted bounding boxes; and performing clustering and fusion processing on the several predicted bounding boxes to obtain bounding boxes for detecting the image of the region to be repaired. The process of establishing a feature pyramid network and performing feature extraction on the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales includes: establishing a three-layer feature pyramid network based on the Faster-RCNN feature extraction network; inputting the preprocessed global image into the first-layer feature pyramid network to obtain the feature extraction prediction results of the first-layer feature pyramid network; inputting the global image processed by the first-layer feature pyramid network into the second-layer feature pyramid network to obtain the feature extraction prediction results of the second-layer feature pyramid network; and inputting the global image processed by the second-layer feature pyramid network into the third-layer feature pyramid network to obtain the feature extraction prediction results of the third-layer feature pyramid network. The process of establishing a feature pyramid network and extracting features from the preprocessed global image based on the feature pyramid network to obtain feature extraction results at various scales further includes: copying the third-layer feature pyramid network to generate a fourth-layer feature pyramid network; performing an upsampling operation on the fourth-layer feature pyramid network to obtain its semantic feature information; performing dimensionality reduction processing on the second-layer feature pyramid network based on a convolutional layer with a kernel size of 1×1 to obtain its high-resolution information; superimposing the semantic feature information of the fourth-layer feature pyramid network and the high-resolution information of the second-layer feature pyramid network to generate a fifth-layer feature pyramid network; performing an upsampling operation on the fifth-layer feature pyramid network to obtain its semantic feature information; performing dimensionality reduction processing on the first-layer feature pyramid network based on a convolutional layer with a kernel size of 1×1 to obtain its high-resolution information; and superimposing the semantic feature information of the fifth-layer feature pyramid network and the high-resolution information of the first-layer feature pyramid network to generate a sixth-layer feature pyramid network. An image cropping module is used to crop the preprocessed global image based on the bounding box to obtain a cropped global image. An image segmentation module is used to establish an image segmentation network for the region to be repaired, and to segment the cropped global image based on the image segmentation network for the region to be repaired to obtain the image of the region to be repaired. The process of establishing a segmentation network for the region to be repaired, and segmenting the cropped global image based on this network to obtain the image of the region to be repaired, includes: performing prediction processing on the cropped global image based on a multi-scale grouping method and the bounding boxes used to detect the region to be repaired, generating a coarse image of the region to be repaired; constructing an FCN segmentation network framework, training the FCN segmentation network framework based on the coarse image of the region to be repaired, obtaining a trained FCN segmentation network framework; performing prediction processing on the coarse image of the region to be repaired based on the trained FCN segmentation network framework, generating a sub-coarse image of the region to be repaired; repeating the above steps until a pixel-level fine image of the region to be repaired is generated. The accuracy assessment module is used to assess the accuracy of the image of the region to be repaired.

Images