A cultural element extraction method based on intelligent identification

Abstract

This application relates to the field of artificial intelligence and discloses a method for extracting cultural elements based on intelligent recognition. It aims to address the problems of low segmentation accuracy, broken or adhered boundaries in cultural heritage images caused by low contrast, complex structure, and scarce annotations. The method includes: acquiring a cultural image and generating an initial coarse mask using pre-trained U-Net++; inputting the image and mask into a conditional generator to generate a high-fidelity reconstructed image; calculating a local adversarial loss using a PatchGAN discriminator; using the gradient of this loss to back-update the U-Net++ segmentation head parameters; iteratively optimizing until the mask converges, and outputting a refined binary mask. This application, through a generative adversarial self-supervised mechanism, significantly reduces the dependence on newly added pixel-level annotations, guiding the model to learn continuous natural boundaries that conform to human cognition, while requiring only a small number of pixel-level annotations for network initialization. The resulting mask can be directly used for vectorization, 3D reconstruction, and digital restoration, supporting high-fidelity digital applications of cultural heritage.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a method for extracting cultural elements based on intelligent recognition. Background Technology

[0002] In the field of digital protection and inheritance of cultural heritage, the ability to automatically extract core cultural elements such as patterns and designs from images of carriers such as murals, ancient books, textiles, and Confucian culture with high precision is a key prerequisite for digital restoration, virtual reproduction, and creative applications.

[0003] However, these images are often difficult to process due to their age, complex manufacturing process, and special medium. First, the images often suffer from fading, staining, or uneven lighting, resulting in low contrast between the target element and the background and blurred edges. This makes it difficult for traditional image segmentation methods to accurately identify the target contour.

[0004] Secondly, cultural elements themselves are often extremely intricate and complex in structure, such as overlapping lines, nested patterns, or artistic gradient transitions. When dealing with these complex structures, general image segmentation models are prone to problems such as boundary breaks, region adhesion, or jagged edges, making it difficult to obtain natural, continuous, and visually perceptible segmentation results.

[0005] More importantly, training high-precision segmentation models requires a large amount of accurate pixel-level labeled data. However, in the field of cultural heritage, labeling is extremely dependent on specialized knowledge, costly, and time-consuming, resulting in a severe shortage of available labeled data. This data bottleneck limits the effectiveness and generalization ability of supervised learning paradigms in this field. Therefore, there is an urgent need for a technical solution that can continuously improve segmentation accuracy through self-supervision with limited labeled data support.

[0006] In summary, existing technologies struggle to effectively extract high-quality cultural element segmentation results from low-contrast, structurally complex cultural heritage images when labeled data is scarce. This has become a prominent technical challenge restricting the high-fidelity digitization of cultural heritage. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a method for extracting cultural elements based on intelligent recognition, which can effectively solve the problems in the background technology. When processing cultural heritage images, existing technologies suffer from weak grayscale differences between target elements and the background due to pigment fading and soiling, resulting in low contrast that significantly weakens the effectiveness of traditional threshold segmentation and edge detection methods. Simultaneously, the highly complex structures of cultural elements, such as overlapping brushstrokes and nested patterns formed by multi-layered drawing, make it difficult for general semantic segmentation models to accurately distinguish adjacent areas, easily producing jagged, broken, or adhered boundaries. Furthermore, high-quality pixel-level annotations are extremely scarce, with high professional knowledge thresholds and huge manual annotation costs, severely restricting the generalization ability and segmentation accuracy of models under the supervised learning paradigm. These shortcomings collectively make it difficult for existing methods to meet the stringent requirements for fine-grained, high-fidelity element extraction in the process of cultural heritage digitization.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for extracting cultural elements based on intelligent recognition, the method comprising the following steps: S1. Obtain the cultural image to be processed and input it into the pre-trained U-Net++ network to generate an initial coarse mask; S2. Input the cultural image and the initial coarse mask into the condition generator to generate a high-fidelity reconstructed image; S3. Calculate the local adversarial loss between the first conditional pair formed by the cultural image and the initial coarse mask, and the second conditional pair formed by the high-fidelity reconstructed image and the initial coarse mask, using the PatchGAN discriminator. S4. Using the gradient of the local adversarial loss, backpropagate and update the parameters of the segmentation head in the U-Net++ network to optimize the initial coarse mask. S5. Repeat steps S2 to S4 to iteratively optimize the mask until the preset convergence condition is met, and output the final fine-grained segmentation mask of cultural elements.

[0009] Furthermore, step S1 generates an initial coarse mask, specifically including: Construct a U-Net++ network with an encoder and decoder structure, where the encoder extracts multi-scale features through multiple sequentially connected downsampling stages, and the decoder reconstructs features through an upsampling path containing multiple nodes and combined with skip connections from the encoder; The output of the top-level node of the decoder is processed by a 1×1 convolutional layer and a Sigmoid activation function to generate a probability map of the same size as the input image as an initial coarse mask.

[0010] Furthermore, the pre-trained U-Net++ network is obtained through the following steps: The U-Net++ network was pre-trained in a supervised manner using a dataset containing images of cultural heritage and their pixel-level annotations. The loss function used during training is a weighted sum of weighted cross-entropy loss and Dice coefficient loss, where the weight of foreground pixels is lower than that of background pixels.

[0011] Furthermore, the condition generator is an encoder-decoder architecture based on residual blocks, in which the encoder is composed of multiple sequentially stacked residual blocks, and the decoder is composed of multiple sequentially transposed convolutional layers, and features corresponding to the resolution in the encoder are fused through skip connections.

[0012] Furthermore, the PatchGAN discriminator is a fully convolutional network that sequentially contains multiple convolutional layers. It is used to receive conditional input consisting of an image and a mask concatenated in the channel dimension, and output a two-dimensional discriminant score map, where each value represents the probability of classifying the corresponding local region in the input tensor as a real image.

[0013] Furthermore, step S4 updates the parameters of the segmentation head in the U-Net++ network, specifically including: During backpropagation, the parameters of the encoder and other nodes of the decoder in the U-Net++ network are frozen, and only the gradient of the local adversarial loss with respect to the segment head parameters is calculated. The segment head parameters are then updated using the optimizer. The segmentation head includes the final feature fusion node at the end of the U-Net++ network decoder and the 1×1 convolutional class layer that follows it.

[0014] Furthermore, the preset convergence conditions in step S5 include: During the iteration process, the change in the intersection-union ratio of the binarized mask obtained in two adjacent iterations is calculated. When the change in the binarized mask is less than the first threshold in multiple consecutive iterations, it is determined to be converged. Alternatively, when the total number of iterations reaches the preset maximum number of rounds, the iteration can be forcibly stopped.

[0015] Furthermore, before outputting the final refined segmentation mask of cultural elements in step S5, the following steps are also included: The probability mask obtained after iterative optimization is subjected to morphological closing operation to fill holes and smooth boundaries, and then binarized by a fixed threshold to generate a fine binary segmentation mask.

[0016] Furthermore, the condition generator integrates an attention mechanism module; The attention mechanism module includes a channel attention module embedded in the encoder residual block and / or a spatial attention module embedded after the transposed convolutional layer of the decoder.

[0017] Furthermore, the discriminator is a multi-scale joint discriminator, which contains multiple parallel discriminative branches. Each branch is a fully convolutional network with different depths and receptive fields, which are used to evaluate the local realism of the image at different scales. The local adversarial loss is the weighted sum of the adversarial losses of each branch output.

[0018] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention constructs a self-supervised optimization framework driven by a generative adversarial network (GAN), organically integrating a segmentation network (U-Net++), a conditional generator, and a discriminator. This method utilizes the discriminator's evaluation of the "realism" of local regions in the generated image to form an adversarial loss signal, which is then used to fine-tune the segmentation network in reverse gradient. This mechanism enables the model to autonomously learn segmentation boundaries that conform to the visual characteristics of cultural heritage, requiring only a small amount of labeled data for network initialization, without adding new annotations. Compared to traditional methods, this effectively overcomes boundary breaks, region adhesion, and jaggedness caused by low image contrast and complex structures, thus obtaining a fine-grained segmentation mask with better continuity, higher accuracy, and greater conformity to human visual perception.

[0019] 2. This invention possesses excellent scalability within its core framework. By introducing an attention mechanism into the conditional generator, the model's ability to capture and reconstruct key feature regions is enhanced, making it particularly suitable for weak-edge scenes such as ink wash patterns and faded areas. Employing a multi-scale joint discriminator simultaneously constrains the realism of the generated image in terms of micro-texture, meso-structure, and macro-layout, thereby ensuring the complete and accurate extraction of complex structures such as nested patterns and overlapping designs. These designs enable the method to be effectively applied to different types of cultural heritage carriers, such as murals, calligraphy and paintings, and unfolded diagrams of artifact surfaces, demonstrating strong generalization ability and scene adaptability. Attached Figure Description

[0020] Figure 1 It is a flowchart of the overall process steps; Figure 2 This is a framework diagram of the method for extracting cultural elements. Detailed Implementation

[0021] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0022] Example 1 Taking the extraction of clothing folds from the murals of Cave 220 at the Mogao Grottoes in Dunhuang as a specific application scenario, this invention details the complete execution flow of its method. The input image is a high-fidelity digital scanned image of a mural with a resolution of 3840×2160 pixels. The image contains pigment fading due to age, localized stains, and overlapping brushstrokes caused by multiple layers of painting. The goal is to accurately extract continuous clothing folds from the image and generate a binary mask with consistent spatial resolution, natural boundaries, and no breaks in the lines.

[0023] The following is in conjunction with the appendix Figure 1-2 The method for extracting cultural elements based on intelligent recognition in this application is described, including the following steps, as detailed below: Step S1: The system first loads the pre-trained U-Net++ network model, which includes the following specific steps: S101. Construct and load the pre-trained U-Net++ network model. S101 aims to construct the main body of the segmentation network with a deeply nested skip connection structure. Its specific architecture and parameter settings are as follows: Encoder section: The encoder consists of five sequentially connected downsampling stages. The input is an RGB three-channel image.

[0024] The first stage involves using a convolutional layer with a kernel size of 3×3 and 64 output channels, followed by batch normalization and ReLU activation to process the input image.

[0025] Downsampling phase: The core operation unit of each phase is repeated twice, each time containing a 3×3 convolutional layer, batch normalization, and ReLU activation function. This unit does not change the spatial size and number of channels of the feature map.

[0026] Channel number rule: After two core operations are completed in each stage, a max pooling layer with a kernel size of 2×2 and a stride of 2 is used to downsample, reducing the feature map space size to half.

[0027] At the start of the second phase, the number of channels in the input feature map is doubled.

[0028] Specifically, the number of feature map channels output in the five stages are 64, 128, 256, 512, and 1024, respectively; the corresponding spatial downsampling ratios relative to the input image are 2, 4, 8, 16, and 32, respectively.

[0029] Decoder section: The decoder contains 5 upsampling paths, each of which terminates at a decoder node.

[0030] Node connection rules: Each decoder node receives and aggregates feature information from multiple layers.

[0031] Specifically, let the decoder nodes be numbered from low to high resolution as follows: to This corresponds to a downsampling factor of 32 to 2. Each node The input sources for (i=1,...,5) include: (a) Output feature map of the encoder at the same level, i.e., the i-th stage; (b) All ratios Deeper, that is, decoder nodes with numbers greater than i. The output feature maps, these feature maps need to be upsampled to the same level as... i has the same spatial resolution.

[0032] Node processing flow: For any node, firstly, the feature maps from all the above sources are concatenated along the channel dimension. Then, the concatenated feature map is processed twice in sequence, each time including a convolutional layer with a kernel size of 3×3, batch normalization, and ReLU activation function, finally outputting the feature map of that node.

[0033] The number of feature map channels output by each node is consistent with the number of output channels of the corresponding encoder stage.

[0034] Upsampling method: When upsampling the feature map of deep nodes to the resolution of shallow nodes, a bilinear interpolation algorithm is uniformly used.

[0035] Output layer: The final output of the network is based on the top-level node of the decoder ( The feature map (with the highest spatial resolution) is then connected to a 1×1 convolutional layer, which maps the number of channels to 1. Finally, it passes through a sigmoid activation function to generate a single-channel probability map with the same size as the input image.

[0036] S102. Pre-train the U-Net++ network using transfer learning on the cultural heritage dataset. To achieve the model's basic segmentation ability for cultural heritage images, the network constructed in S101 is pre-trained under supervision.

[0037] This embodiment uses the publicly available ICDAR2019 cultural heritage image dataset, which contains various cultural relic images and their corresponding pixel-level annotations. The specific pre-training process is as follows: Data preprocessing: All images and corresponding annotations in the dataset are scaled to a fixed size, and the RGB pixel values ​​of the images are normalized to the range [0,1].

[0038] Training settings: Using the Adam optimizer, the initial learning rate is set to 0.001, and the momentum parameters β1=0.9 and β2=0.999. The batch size is set to 8. The number of training epochs is set to 100.

[0039] Loss function calculation: In each iteration, the loss between the network prediction result and the ground truth label is calculated using weighted cross-entropy loss. and Dice coefficient loss Linear combinations constitute the structure, and the calculation formula is:

[0040] In the weighted cross-entropy loss, to alleviate the problem of sparse target pixels of cultural heritage, the weight of the target cultural element in the foreground pixel is explicitly set to 1, and the weight of the background pixel is set to 5.

[0041] Dice coefficient loss It is used to directly maximize the overlap between the predicted region and the real region.

[0042] Parameter update: Calculate the loss for the current batch. The gradient is calculated using the backpropagation algorithm, and the Adam optimizer is used to update all trainable parameters of the U-Net++ network.

[0043] Training complete: When the model's loss on the validation set no longer decreases significantly or the preset number of training rounds is reached, save the network parameters to obtain the pre-trained U-Net++ model.

[0044] S103. Generate an initial coarse mask using the pre-trained model. This step, S103, follows from S101 and S102, aiming to input the specific cultural image to be processed into a model that already possesses basic segmentation capabilities, obtaining a preliminary probabilistic segmentation result that can be used for subsequent optimization. The specific operations are as follows: The input image from which the cultural elements are to be extracted, such as the "Flying Apsaras" mural image in Example 1, is preprocessed. The preprocessing method is consistent with the preprocessing method of the training data in S102, including scaling the image to the same fixed size as during pre-training and performing the same pixel value normalization ([0,1]).

[0045] The preprocessed image data is input into the pre-trained U-Net++ model obtained in S102. The model performs forward propagation calculations and finally outputs a single-channel probability map with the same size as the network input through the Sigmoid activation function of the output layer. The value of each pixel in this map is a floating-point number between 0 and 1, representing the confidence that the pixel belongs to the target cultural element (foreground).

[0046] The aforementioned probability map is used as the initial coarse mask. Here, "coarse mask" can refer directly to the probability map, or it can be binarized by setting a threshold, such as 0.5, to obtain a binary mask. To preserve maximum information for subsequent optimization steps, this embodiment preferably uses the unthresholded probability map itself as the initial coarse mask output.

[0047] After completing the above steps, the conditional generator G and discriminator D need to be pre-trained to enable them to perform basic image reconstruction and discrimination. The same cultural heritage dataset is used for training, which contains images and their corresponding real pixel-level annotations (masks).

[0048] 1. Pre-training of the condition generator G: Input: Real image x and real mask Concatenate them into a 4-channel tensor along the channel dimension.

[0049] Output: Reconstructed image .

[0050] Loss function: L1 reconstruction loss .

[0051] Objective: To enable G to reconstruct textures and structures consistent with real images based on real masks.

[0052] 2. Pre-training of discriminator D: Input: Two sets of 4-channel tensors, respectively and .

[0053] Output: Discriminant score image.

[0054] Loss function: A binary cross-entropy loss is used to enable D to accurately determine whether the input is a real image.

[0055] Objective: To enable D to give high confidence for "real image + real mask" and low confidence for "generated image + real mask".

[0056] After pre-training, the parameters of the condition generator G and the discriminator D are fixed. During the iterative optimization process of this method, the condition generator and the discriminator are used as pre-trained fixed modules, and their parameters are not updated during the iteration process. They are only used to provide adversarial gradient signals, thereby ensuring the stability and repeatability of the optimization process.

[0057] For step S2, based on step S1, an initial coarse mask containing preliminary location of the target area but with breaks and misjudgments has been obtained. Step S2 uses this mask and the original image to reconstruct a high-fidelity image of the target area that is visually continuous, realistic, and consistent with the material and painting style of the cultural heritage through a conditional generation network.

[0058] The reconstructed image will serve as the basis for subsequent discriminator evaluation of "realism," thereby driving mask optimization. The specific implementation is as follows: S201. Construct the input tensor of the condition generator G. Concatenate the original cultural image's RGB three channels with the initial coarse mask single-channel probability map obtained in step S103 along the channel dimension to form a 4-channel tensor.

[0059] This tensor integrates the original visual information with the preliminary segmentation information, serving as the conditional input to the generator.

[0060] S202. Construct and run the encoder portion of the condition generator G. Generator G employs an encoder-decoder architecture. Its encoder consists of nine consecutive stacked residual blocks.

[0061] General structure of residual blocks: The specific structure of each residual block is as follows: The input feature map first passes through a convolutional layer with a kernel size of 3×3, a stride of 1, and padding of 1, followed by batch normalization and the ReLU activation function; Then, it goes through a second layer with the exact same structure: "3×3 convolution + batch normalization + ReLU".

[0062] Meanwhile, the residual block retains an identity mapping connection, which directly adds the block's input to the output of the second convolutional layer element by element, and then outputs the sum after passing it through a ReLU activation function.

[0063] Downsampling and channel number rules: The encoder achieves downsampling by setting the stride to 2 in the first convolution of a specific residual block and adjusting the padding accordingly to ensure that the size calculation is correct.

[0064] Specifically, the first and second residual blocks maintain the input resolution, with 64 input and 64 output channels each. Starting from the third residual block, the stride of the first convolutional layer in each block is set to 2 to achieve downsampling, halving the spatial size, while doubling the number of output channels in that layer.

[0065] Therefore, after processing the 9 residual blocks, the spatial size of the feature map gradually decreases, with the channel number changing in the following sequence: 64→64→128→256→512→512→512→512→512. Finally, the encoder outputs a deep semantic feature map with 512 channels.

[0066] S203. Construct the decoder portion of the conditional generator G and perform image reconstruction. The decoder is responsible for upsampling the deep features output by the encoder and reconstructing them into an image. It consists of eight transposed convolutional layers in sequence.

[0067] Upsampling process: The kernel size of each transposed convolutional layer is 4×4 with a stride of 2, which expands the spatial size of the input feature map to twice its original size. The number of output channels of the 8 layers are set as follows: 256, 128, 64, 32, 16, 8, 4, 3.

[0068] Except for the last transposed convolutional layer, each layer is followed by batch normalization and ReLU activation functions.

[0069] Furthermore, at each upsampling level, the decoder uses skip connections to concatenate the feature map with the same spatial resolution from the encoder and the upsampled feature map at this level along the channel dimension. This process fuses the high-frequency details retained by the encoder, thereby more accurately restoring image details.

[0070] The final transposed convolutional layer outputs a 3-channel feature map, and the pixel values ​​are constrained to the [-1, 1] interval using the Tanh activation function. These values ​​are then linearly mapped to the [0, 1] interval in subsequent processing to obtain the final high-fidelity reconstructed RGB image. The image size is designed to be consistent with the original input image.

[0071] Step S2 completes the conversion from the original image + coarse mask to a high-fidelity reconstructed image. The condition generator G acts as an intelligent inpainting unit: specifically, it receives the original scene with defect hints and attempts to output a realistic and natural version, in which textures, brushstrokes, and color transitions missing due to low contrast or breaks in the foreground areas indicated by the mask are reasonably generated or filled in.

[0072] For example, in this embodiment, the broken lines in the "flying apsara's" clothing are reconstructed as continuous lines. The quality of the generated image, i.e., its consistency with the materials and style of the authentic cultural heritage, will become a key indicator for evaluating the rationality of the initial mask.

[0073] Next, the system will introduce a discriminator D to quantitatively evaluate this "authenticity" by comparing the real original image with the generated image, thereby generating a gradient signal that drives mask optimization.

[0074] For step S3, based on step S2, a discriminator D is constructed. By comparing the two sets of input conditions, "real image + mask" and "generated image + same mask," the realism of the generated image is evaluated at the local image patch level, and the adversarial loss is calculated accordingly. This loss quantifies the difference between the current generated result and the texture and style of real cultural heritage, providing a clear gradient direction for subsequent optimization of the segmentation mask. The specific implementation steps are as follows: S301, the discriminator D adopts a fully convolutional network architecture. Its design goal is to receive a 4-channel conditional image, which is composed of the image and the mask, and output a two-dimensional discriminant score map.

[0075] Input layer: The discriminator input is a 4-channel tensor, whose channel configuration is consistent with the input of the generator G in S201.

[0076] Convolutional layer sequence: The network contains 5 consecutive convolutional layers.

[0077] Layer 1: The convolution kernel size is 4×4, the stride is 2, and the number of output channels is set to 64. It is followed by the LeakyReLU activation function, with its negative slope parameter α set to 0.2.

[0078] Layer 2: The convolution kernel size is 4×4, the stride is 2, and the number of output channels is set to 128. This is followed by instance normalization and the LeakyReLU activation function, α=0.2.

[0079] Layer 3: The kernel size is 4×4, the stride is 2, and the number of output channels is set to 256. This is followed by instance normalization and the LeakyReLU activation function α=0.2.

[0080] Layer 4: Convolutional kernel size is 4×4, stride is 1, and output channel count is set to 512. This is followed by instance normalization and the LeakyReLU activation function α=0.2.

[0081] Layer 5: The kernel size is 4×4, the stride is 1, and the number of output channels is set to 1.

[0082] Output layer: After the 5th convolutional layer, no normalization or activation function is applied; a single-channel feature map is directly output. Each pixel value on this feature map represents the probability that the discriminator classifies a region of approximately 70×70 pixels in the input tensor as "real" in this architecture. The entire network does not contain any fully connected layers.

[0083] S302. In each iteration, the discriminator D needs to process two sets of inputs for comparison.

[0084] The first set of input true condition pairs: concatenate the original cultural image (x) with the initial coarse mask (m) used in the current iteration along the channel dimension to obtain the condition tensor [x,m]. This tensor represents "the representation of the true image under the given mask conditions".

[0085] The second set of input generation conditions: the high-fidelity image output by generator G ( The conditional tensor is obtained by concatenating the initial coarse mask (m) with the same initial coarse mask (m) along the channel dimension. This tensor represents "the rendering of the generated image under the same mask conditions".

[0086] The discrimination process involves inputting the two sets of 4-channel tensors into the discriminator D constructed by S301 for forward propagation. Discriminator D outputs a corresponding discrimination score map for each input tensor. and .

[0087] S303. Based on the discrimination results obtained in S302, calculate the adversarial loss used to drive the optimization of the generator and the segmentation network.

[0088] Loss function definition: Conditional adversarial loss is adopted, its basic form being binary cross-entropy. For a batch of training samples, local adversarial loss is used. Calculate using the following formula:

[0089] Where E represents the expected value, which is approximated in practice by calculating the average loss of all samples in the current training batch. D(x,m) is the discriminant score of the true conditional pair. It is the discrimination score of the generated condition pairs.

[0090] The first term of the formula encourages the discriminator to correctly classify the true condition pair as "true", with a score close to 1; The second item encourages the discriminator to correctly classify the generated condition pair as "false", with a score close to 0.

[0091] This loss function does not include L1 or L2 pixel-level reconstruction loss terms to avoid overly smoothing the generated image, thereby better preserving the unique graininess, smudges, and other complex high-frequency textures in cultural heritage images.

[0092] Training objective description: During the pre-training phase (as described in the "Pre-training of Conditional Generator and Discriminator" section above), the discriminator D and generator G are optimized according to the standard adversarial training paradigm, that is, the objective of discriminator D is to maximize the aforementioned loss. To better distinguish between true and false, the generator G aims to minimize... To generate realistic images.

[0093] However, in the core closed-loop iterative optimization stage of this invention (i.e., steps S2 to S5), as clarified in steps S401 and S402, the network parameters of the pre-trained generator G and discriminator D remain frozen and do not participate in parameter updates during this stage. They serve only as fixed forward computation modules used to calculate the local adversarial loss. It also provides gradient signals. The sole optimization objective at this stage is to utilize this loss. The gradient is used to update only the segmentation head parameters of the U-Net++ network (as described in step S4) to drive the initial coarse mask to continuously optimize in a direction that guides the generation of more realistic images. This self-supervised mechanism of "fixed G / D, optimizing only the segmentation head" is the key to the solution's ability to improve segmentation accuracy without adding new labeled data.

[0094] Through steps S301 to S303, an image authenticity assessment and feedback mechanism based on local perception is established. The discriminator D does not care about the global consistency of the image, but focuses on judging whether each 70×70 local region presents the true texture and structure of cultural heritage under a given mask condition.

[0095] The calculated adversarial loss This quantifies the local distribution differences between the generated image and the real image. The difference signal indicates inconsistencies in the visual reconstruction of the region defined by the current mask.

[0096] Next, the system will use this signal to update the U-Net++ segmentation network that generated the initial mask through backpropagation, thereby guiding the mask to evolve in a direction that can generate more realistic images, that is, learning more accurate element segmentation results with more natural boundaries.

[0097] Step S4 uses this loss to critique the generated image, and then reverse-optimizes the U-Net++ network that initially produced the coarse mask, rather than the generator G itself. The specific update strategy and steps are as follows: S401. Construct a joint computation graph and calculate the gradient of the adversarial loss with respect to the U-Net++ parameters.

[0098] In this step, the parameters of the condition generator G and the discriminator D remain fixed (i.e., they do not participate in this round of gradient update), serving only as fixed modules for forward computation and gradient propagation. The system constructs the following computation graph to calculate the gradient: 1. Forward computation sequence: U-Net++ receives input images Output the initial coarse mask for the current iteration. .

[0099] The condition generator G receives the condition input. (Image and mask are stitched together along the channel dimension), outputting the reconstructed image. .

[0100] Discriminator D receives the true condition pairs respectively With generation conditions Output the corresponding discrimination score map and calculate the local adversarial loss accordingly. .

[0101] 2. Gradient backpropagation: In the automatic differentiation framework, from loss Initiate backpropagation.

[0102] The parameters of the discriminator D are frozen in this iteration, so the gradient is not calculated or updated.

[0103] The parameters of generator G are: to participate in gradient calculation to ensure that the gradient flow can continue to be propagated, but not to be updated in subsequent optimization steps (i.e., only to serve as a "gradient channel").

[0104] The parameters of U-Net++: the gradient continues to be passed to U-Net++, and the calculation... The gradient of its parameters.

[0105] S402. Strictly limit the gradient update range to the "segmentation head" part of U-Net++.

[0106] To maintain the stability of pre-trained features and achieve accurate optimization, only the "segmentation head" parameters of U-Net++ are updated after backpropagation, while all other parameters are frozen. 1. Frozen section: Parameters of all layers in the U-Net++ encoder; The parameters of all nodes in the decoder except for the "segmentation head".

[0107] 2. Updatable part (segment header): This refers to all trainable layers after the topmost node of the U-Net++ decoder, specifically including: The convolutional layer parameters corresponding to the two "3×3 convolution + batch normalization + ReLU" operations within this node; The weights and biases of the 1×1 convolutional layer follow immediately.

[0108] Implementation method: Only the segmentation head parameters mentioned above are registered in the optimizer; encoder and other decoder parameters are not registered.

[0109] S403. Robustly optimize and update the segmentation head parameters; After obtaining the gradients of the segmentation head parameters, perform the following optimization steps: 1. Gradient clipping: Calculate the L2 norm of the gradients of all segmentation head parameters. If the norm exceeds the threshold of 1.0, scale the gradient proportionally to make its norm equal to 1.0.

[0110] 2. Parameter update: Update the segmentation head parameters using the Adam optimizer, with the optimizer settings as follows: Learning rate ; First-order moment attenuation coefficient ; Second moment attenuation coefficient .

[0111] 3. Generator and discriminator parameter states: In this optimization step, the parameters of the condition generator G and the discriminator D are not updated and remain in their pre-trained fixed state.

[0112] Step S4 completes the most innovative step in the generative adversarial feedback loop: transforming the discriminator's judgment of "image realism" into a precise optimization signal for the "segmentation mask generator" U-Net++. The entire mechanism can be understood as follows: if the discriminator determines the image generated based on the current mask to be "fake," it means that the mask has failed to correctly define the region that allows the generator to reconstruct realistic textures. Therefore, the system, through backpropagation, slightly adjusts the parameters of the U-Net++ segmentation head so that it can generate a slightly different mask for the same input image next time. The goal of this new mask is to guide the generator to produce a more "realistic" image. At this point, a single iteration of the optimization loop is complete.

[0113] Finally, for step S5, the iterative loop of the entire self-supervised optimization process is managed, a clear stopping criterion is set, and the mask generated after optimization is post-processed as necessary. The final output is a fine-grained segmentation result that can be directly applied to downstream tasks. The specific process breakdown is as follows: S501. The system starts with the initial coarse mask generated in S103 and enters a repeatable optimization loop.

[0114] Definition of a single loop: A single iteration loop refers to the entire process of sequentially executing S201 to S203, condition generation, S301 to S303, adversarial discrimination and loss calculation, and S401 to S403, the entire process of updating the segment head gradient.

[0115] Looping execution: The system will repeatedly execute the above iterative loop. At the end of each loop, denoted as the k-th iteration, a new mask generated by the updated U-Net++ network is obtained. .

[0116] S502. To determine when to stop the optimization, it is necessary to quantitatively evaluate the degree of mask improvement between adjacent iterations.

[0117] Change Calculation: After the k-th iteration (k≥2), calculate the Intersection over Union (IoU) between the current generation mask m_k and the previous generation mask m_{k-1}. Specifically, first, binarize the two probability masks using a fixed threshold of 0.5, then calculate the intersection pixel count of the two binarized masks divided by the union pixel count to obtain the IoU value. The mask change is defined as... That is, the absolute difference between the IoU values ​​of the masks in adjacent rounds.

[0118] Convergence Criteria Determination: The system presets two termination conditions; iteration stops when either condition is met. Condition A, stable convergence: If the mask change ΔIoU in three consecutive iterations is less than the threshold 0.001, then the mask is considered stable and the optimization has converged.

[0119] Condition B, Maximum number of iterations: If the total number of iterations reaches the preset upper limit of 10 iterations, the iteration will be forcibly stopped to prevent overfitting or invalid calculations.

[0120] The system checks the above conditions after each iteration. In this embodiment, after the 5th iteration, the system stops because condition A is met, assuming that the ΔIoU of the 3rd, 4th, and 5th iterations are all <0.001. At this time, the IoU of the mask increases to 0.89.

[0121] S503. After the iteration terminates, the last generation mask is processed to improve its visual quality and engineering usability.

[0122] Morphological closing operation: for the final probability mask such as Perform morphological closing operations. Specifically, first, dilate the mask using a 5x5 pixel circular structuring element, then erode it using the same structuring element. The purpose of this operation is to fill in small holes and breaks in the mask and smooth rough boundaries, especially for minor discontinuities in cultural heritage images caused by noise or weak contrast.

[0123] Generating the final binary mask: The probability mask, after closing operations, is binarized using a threshold of 0.5. Specifically, pixels with a probability value greater than or equal to 0.5 are assigned a value of 1 (foreground, representing cultural elements), and pixels with a probability value less than 0.5 are assigned a value of 0 (background). The result is a binary image with the same spatial resolution as the original input image, i.e., a fine-grained segmentation mask.

[0124] Step S5 ensures that the optimization process can automatically stop after the mask quality is significantly improved, balancing efficiency and effectiveness. Thus, the cultural element extraction method based on intelligent recognition has completed the fully automated extraction process from an original cultural image to a refined, binary segmentation mask.

[0125] Example 2 To verify the generalization ability of this invention on different types of cultural heritage carriers and to demonstrate its scalability at the generator architecture level, this embodiment takes the extraction of landscape patterns from a section of the Song Dynasty painting "A Panorama of Rivers and Mountains" as an example. To address the problem of extremely blurred target boundaries caused by ink wash effects, this embodiment introduces an attention mechanism into the conditional generator for enhancement.

[0126] Application Scenario and Initialization: The input image is a partial digital scan image of the painting "A Panorama of Rivers and Mountains" with a resolution of 2560×1440 pixels. The contours of the mountains and rocks are gradually blurred due to the ink wash rendering, resulting in extremely low contrast with the background water. In the initial segmentation step (corresponding to S1), an initial coarse mask is generated using the same pre-trained U-Net++ network and process as in S101 to S103 of Example 1. Due to the blurred boundaries, the initial mask is severely diffused in the mountain edge region, and its IoU with expert annotations is only 0.58.

[0127] Construction and execution of the improved condition generator G' (corresponding to a variant of step S2): This embodiment employs a conditional generator G' that integrates an attention mechanism. Its overall encoder-decoder framework and input / output format are consistent with the generator G in Embodiment 1 (see S201, S202, S203), the core difference being the embedding of the attention module: Before the identity mapping connection of each residual block (9 in total) in the encoder section and the final ReLU activation, a channel attention module is embedded. This module performs the following operations in sequence: Global average pooling is performed on the feature map input to this module, compressing the two-dimensional features of each channel into a scalar.

[0128] The compressed vector is input into a two-layer fully connected network. The first fully connected layer compresses the number of channels to 1 / 16 of the original (i.e., a compression ratio of 16) and uses the ReLU activation function; the second fully connected layer restores the number of channels to the original number and uses the Sigmoid activation function, outputting a channel weight vector between 0 and 1.

[0129] The channel weight vector obtained above is multiplied element-wise with the original input feature map of the module along the channel dimension, thereby recalibrating the feature importance of each channel.

[0130] After upsampling and performing skip connections with encoder features at each transposed convolutional layer in the decoder section, and before proceeding to the next transposed convolutional layer or the final output layer, a spatial attention module is embedded. This module performs the following operations sequentially: The input feature map is subjected to two convolution operations. The first convolution uses a 1×1 kernel, and the number of output channels is halved; the second convolution uses a 1×1 kernel, and the number of output channels is 1. After both convolutions, batch normalization and ReLU activation function are applied (ReLU is optional after the second convolution).

[0131] For the single-channel feature map output by the second convolution, the Sigmoid activation function is used to generate a spatial weight map with the same spatial size as the input feature map and each pixel value in the interval [0, 1].

[0132] The generated spatial weight map is multiplied element-wise with the original input feature map that has not been processed by this module, thereby highlighting the feature responses of important spatial regions.

[0133] The generator G', which integrates the aforementioned attention module, receives the same 4-channel conditional input (image + mask) and outputs a high-fidelity reconstructed image. Its reconstruction focuses on simulating the granular crystalline texture of mineral pigments and the natural gradations of ink wash in blue-green landscape paintings.

[0134] Adversarial discrimination, gradient update, and iterative convergence (corresponding to S3, S4, S5): The same PatchGAN discriminator D and its loss calculation method as in S301 to S303 of Example 1 are used. The discriminator focuses on judging the local realism of pigment texture and smudging transition in the generated image.

[0135] The gradient backpropagation strategy (updating only the U-Net++ segmentation head), gradient pruning threshold (1.0), and optimizer parameters (Adam, lr=0.0002, β1=0.5, β2=0.999) are exactly the same as those in S401 to S403 of Example 1.

[0136] The same iterative convergence conditions as S501 to S503 in Example 1 were used (ΔIoU < 0.001 for 3 consecutive rounds or total number of rounds = 10). This example converged after the 6th iteration, and the final mask IoU reached 0.85. Before binarization of the final mask, the structuring element used in the morphological post-processing was adjusted to a 3×3 elliptical structuring element to better adapt to the streamlined characteristics of the landscape outline and avoid the boundary blunting that might be caused by using a circular kernel.

[0137] The final generated fine-grained binary mask successfully separated the main body of the mountains and rocks, with natural and continuous boundaries, effectively overcoming the blurring problem caused by ink wash. The mask completely preserves the structural information of the landscape pattern and can be used for subsequent tasks, such as transferring the blue-green landscape style of "A Panorama of Rivers and Mountains" to other digital paintings, ensuring the accuracy of the core pattern structure during style transfer.

[0138] Example 3 To further verify the applicability of this invention to the surface unfolding of three-dimensional objects, such as carriers of diverse cultural heritage and Confucian culture, and to demonstrate its flexibility at the discriminator architecture level, this embodiment adopts a multi-scale joint discrimination strategy for the task of extracting the underglaze cobalt blue pattern from a blue and white porcelain vase.

[0139] The input data is a 360-degree circular digital image of a Ming Dynasty blue-and-white porcelain vase, with a resolution of 4096×1024 pixels, featuring a lotus scroll pattern. Due to the glaze layer and the influence of ancient firing techniques, some cobalt blue lines are broken and blurred. In the initial segmentation step (corresponding to S1), an initial coarse mask is generated using the same pre-trained U-Net++ network and process as in S101 to S103 of Example 1. The initial mask shows obvious breaks at the tips of the lotus petals and the junctions of the vines, with an IoU of 0.60 with the reference label.

[0140] Conditional generation and multi-scale adversarial discrimination, corresponding to S2 and S3 variants: Condition Generator: The condition generator G in this embodiment adopts the same encoder-decoder architecture (based on ResNet-block) as described in S202 and S203 of Embodiment 1, but does not embed an attention mechanism. Its input and output formats are consistent with those in S201 of Embodiment 1.

[0141] Construction and operation of the multi-scale PatchGAN discriminator: The key improvement lies in the discriminator. This embodiment constructs a network containing three parallel discriminative branches.

[0142] The three branches (denoted as D1, D2, and D3) have similar but different depths of fully convolutional network structures, and their design is based on the discriminator D (S301) in Example 1. Specifically, as follows: D1 (small scale): Consists of 3 convolutional layers to achieve a receptive field of approximately 16×16 pixels. Each layer has a 4×4 kernel with strides of 2, 2, and 1 respectively. The number of output channels is 64, 128, and 1 respectively. Only the last layer outputs a single-channel feature map.

[0143] D2 (Medium Scale): Consists of 4 convolutional layers to achieve a receptive field of approximately 32×32 pixels. Each layer has a 4×4 kernel with strides of 2, 2, 1, and 1 respectively. The number of output channels is 64, 128, 256, and 1 respectively. Instance normalization (same as S301) is used after layers 2 and 3.

[0144] D3 (large scale): Its structure and parameters are completely consistent with the discriminator D defined in Example 1 S301 (5 layers, 70×70 receptive field).

[0145] For each iteration, the generated condition pair [ŷ, m] and the true condition pair [x, m] are input into the three branches of the discriminator, respectively. Each branch independently outputs its discriminant score map and calculates its adversarial loss according to the same formula defined in Example 1, S303. , , (The average of the scores for each branch is calculated separately.)

[0146] Total adversarial loss: The final total adversarial loss used to drive model optimization is a weighted sum of the losses of the three branches. This design forces the generator to simultaneously ensure the realism of microscopic textures (cobalt particles, glaze), mesoscopic brushstrokes, and macroscopic composition during reconstruction.

[0147] Gradient update, iteration, and post-processing (corresponding to S4 and S5): The gradient backpropagation strategy (updating only the U-Net++ segmentation head), gradient pruning threshold (1.0), and optimizer parameters (Adam, lr=0.0002, β1=0.5, β2=0.999) are adopted exactly as in Examples 1 (S401 to S403). Gradient calculation is based on the total loss. .

[0148] The same iterative convergence conditions as S501 and S502 in Example 1 are used. In this example, the convergence condition is met after the 7th iteration, and the final mask IoU reaches 0.87.

[0149] Before final mask binarization, morphological closing operations are performed. A 3×3 pixel cross-shaped structuring element is used, defined as follows: the center pixel and its four directly adjacent pixels (above, below, left, and right) are the foreground (value 1), and the remaining corner pixels are the background (value 0). This shape effectively connects broken lines in the horizontal and vertical directions while avoiding excessive expansion of diagonal lines.

[0150] The final generated fine-grained binary mask clearly restored the complete geometric structure of the lotus scroll pattern, successfully repairing the line breaks caused by manufacturing defects. This mask can be directly used for high-precision digital archiving of porcelain patterns and provides an accurate outline basis for 3D printing, relief reproduction, or digital derivative designs based on the pattern.

[0151] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

[0152] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0153] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for extracting cultural elements based on intelligent recognition, characterized in that, Includes the following steps: S1. Obtain the cultural image to be processed and input it into the pre-trained U-Net++ network to generate an initial coarse mask; S2. Input the cultural image and the initial coarse mask into the condition generator to generate a high-fidelity reconstructed image; S3. Calculate the local adversarial loss between the first conditional pair formed by the cultural image and the initial coarse mask, and the second conditional pair formed by the high-fidelity reconstructed image and the initial coarse mask, using the PatchGAN discriminator. S4. Using the gradient of the local adversarial loss, backpropagate and update the parameters of the segmentation head in the U-Net++ network to optimize the initial coarse mask. S5. Repeat steps S2 to S4 to iteratively optimize the mask until the preset convergence condition is met, and output the final fine-grained segmentation mask of cultural elements.

2. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, Both the condition generator and the PatchGAN discriminator are obtained through pre-training, and their network parameters remain fixed during iterative optimization, only used to calculate local adversarial loss and provide gradient signals. The pre-training objective of the conditional generator is to reconstruct a high-fidelity image that is consistent with the real image in terms of texture and structure, given a real image and a real mask. The pre-training objective of the discriminator is to accurately distinguish the local authenticity of "real image + real mask" and "generated image + real mask".

3. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, Step S1 generates an initial coarse mask, specifically including: Construct a U-Net++ network with an encoder and decoder structure, where the encoder extracts multi-scale features through multiple sequentially connected downsampling stages, and the decoder reconstructs features through an upsampling path containing multiple nodes and combined with skip connections from the encoder; The output of the top-level node of the decoder is processed by a 1×1 convolutional layer and a Sigmoid activation function to generate a probability map of the same size as the input image as an initial coarse mask.

4. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, The pre-trained U-Net++ network is obtained through the following steps: The U-Net++ network was pre-trained in a supervised manner using a dataset containing images of cultural heritage and their pixel-level annotations. The loss function used during training is a weighted sum of weighted cross-entropy loss and Dice coefficient loss, where the weight of foreground pixels is lower than that of background pixels.

5. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, The condition generator is an encoder-decoder architecture based on residual blocks. Its encoder consists of multiple sequentially stacked residual blocks, and its decoder consists of multiple sequentially transposed convolutional layers. Features corresponding to the resolution in the encoder are fused through skip connections. The condition generator integrates an attention mechanism module, which includes a channel attention module embedded in the encoder residual block and / or a spatial attention module embedded after the transposed convolutional layer of the decoder.

6. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, The PatchGAN discriminator is a fully convolutional network containing multiple convolutional layers. It receives conditional input consisting of an image and a mask concatenated along the channel dimension and outputs a two-dimensional discriminant score map. Each value represents the discriminant's score for classifying a corresponding local region in the input tensor as a real image. This score can be mapped to a probability value after passing through the Sigmoid function.

7. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, Step S4 involves updating the parameters of the segmentation head in the U-Net++ network, specifically including: During backpropagation, the parameters of the encoder and other nodes of the decoder in the U-Net++ network are frozen, and only the gradient of the local adversarial loss with respect to the segment head parameters is calculated. The segment head parameters are then updated using the optimizer. The segmentation head includes the final feature fusion node at the end of the U-Net++ network decoder and the 1×1 convolutional class layer that follows it.

8. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, The preset convergence conditions in step S5 include: During the iteration process, the change in the intersection-union ratio of the binarized mask obtained in two adjacent iterations is calculated. When the change in the binarized mask is less than the first threshold in multiple consecutive iterations, it is determined to be converged. Alternatively, when the total number of iterations reaches the preset maximum number of rounds, the iteration can be forcibly stopped.

9. The method for extracting cultural elements based on intelligent recognition according to claim 1, characterized in that, Before outputting the final fine-grained segmentation mask of cultural elements in step S5, the following steps are also included: The probability mask obtained after iterative optimization is subjected to morphological closing operation to fill holes and smooth boundaries, and then binarized by a fixed threshold to generate a fine binary segmentation mask.

10. The method for extracting cultural elements based on intelligent recognition according to claim 6, characterized in that, The discriminator is a multi-scale joint discriminator, which contains multiple parallel discriminant branches. Each branch is a fully convolutional network with different depths and receptive fields, which are used to evaluate the local realism of the image at different scales. The local adversarial loss is the weighted sum of the adversarial losses of each branch output.

Images