An automatic extraction method of heritage typical scenes based on user-generated images

Abstract

The present application belongs to the technical field of cultural heritage digitization, and particularly relates to a typical heritage scene automatic extraction method based on user-generated images, comprising: performing image background cutting and size normalization preprocessing on each user-generated image of a target heritage point; extracting the global visual feature vector of the preprocessed image and performing normalization processing thereon; calculating the cosine similarity between images based on the normalized global visual feature vector of each image; identifying non-noise scene clusters using a density clustering algorithm; calculating the cluster size, intra-cluster average similarity and center image representativeness of each non-noise scene cluster based on the cosine similarity between images, so as to determine the comprehensive score of the scene cluster; recording the scene cluster whose cluster size, intra-cluster average similarity and center image representativeness all meet the corresponding index preset threshold as an effective cluster; and selecting a preset number of typical scene clusters from the effective clusters. The present application can realize the full-process automatic extraction of typical heritage scenes.

Description

Technical Field

[0001] This invention belongs to the technical field of cultural heritage digitization, and more specifically, relates to a method for automatically extracting typical heritage scenes based on user-generated images. Background Technology

[0002] With the development of mobile internet, social media, and digital imaging devices, the public often shares images related to heritage sites, historic districts, and traditional villages on social media. These user-generated images reflect the public's actual viewing angles of heritage spaces, repeatedly photographed subjects, and key points of scene perception, thus serving as an important source for understanding the public's perception of heritage scenes.

[0003] Existing technologies have already explored related areas such as cultural heritage image processing, social media image landscape analysis, and image clustering. For example, Chinese patent CN110781253A discloses a method for creating a database based on historical and cultural heritage image data and constructing an image feature model for feature extraction; Chinese patent CN113159122B discloses a method for extracting urban landscape vectors using depth visual features and measuring the overall landscape differences and characteristics between cities accordingly; and Chinese patent CN106055624A discloses a method for processing and grouping photos of the same scene or the same subject to improve the efficiency of mobile terminal photo album organization and browsing.

[0004] However, user-generated images are characterized by scattered sources, inconsistent shooting perspectives, varying image sizes and formats, and inconsistent quality. Traditional methods of organizing heritage images often rely on manual screening and classification, making it difficult to efficiently identify typical scenes of heritage sites within large-scale image data. Existing image clustering and semantic segmentation methods typically only yield a few similar image sets, making it difficult to determine whether these image sets possess a stable common visual framework, and also difficult to extract compositional focal points, stable structural regions, and semantic element distributions from the image sets. For typical heritage scenes, it is even more necessary to statistically analyze the stable frequency and spatial distribution of semantic elements across multiple images within a unified spatial coordinate system.

[0005] Therefore, there is an urgent need for a technical solution that can generate image data for users and automatically complete image normalization and database construction, visual feature extraction, typical scene clustering and screening, geometric registration, overlay representation, structural hotspot extraction and semantic frequency mapping, so as to improve the automation, stability and interpretability of typical scene extraction of heritage sites. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an automatic extraction method for typical heritage scenes based on user-generated images, the purpose of which is to achieve full-process automated extraction of typical heritage scenes.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for automatically extracting typical heritage scenes based on user-generated images is provided, comprising: Obtain the user-generated image set of the target heritage site and construct the corresponding basic index table. Each index includes the image ID of the user-generated image and its corresponding width, height and aspect ratio. Perform image background cropping and size normalization preprocessing on the user-generated image corresponding to each image ID. Each preprocessed image is pixel-normalized and then input into the visual base model to obtain the global visual feature vector of the preprocessed image, which is then normalized. The cosine similarity between images is calculated based on the normalized global visual feature vectors of each image. A density clustering algorithm is used to identify non-noise scene clusters based on the minimum number of images contained in a cluster adaptively preset for the target heritage point and the minimum number of images used to judge the density core region. Based on the cosine similarity between images, the cluster size, average similarity within the cluster, and representativeness of the center image are calculated for each non-noise scene cluster to determine the overall score of the scene cluster. The center image is the image with the highest average cosine similarity with other member images within the cluster, and the representativeness of the center image is the value of the highest average cosine similarity. Scene clusters whose cluster size, average intra-cluster similarity, and central image representativeness all meet the preset thresholds of the corresponding indicators are recorded as valid clusters; a preset number of typical scene clusters are selected from the valid clusters: scene clusters whose average intra-cluster similarity and central image representativeness both reach the preset thresholds of the corresponding indicators are regarded as typical scene clusters; the remaining number of typical scene clusters are sorted according to the comprehensive score and the highest score is retained to complete the extraction of typical heritage scenes.

[0008] Furthermore, the preprocessing includes: performing EXIF ​​orientation correction on the user-generated image and uniformly converting the corrected image to RGB format; estimating the background color based on the four corner pixels of the RGB converted image, identifying and cropping the solid color border; normalizing the width of the non-background area image while maintaining the aspect ratio of the cropped non-background area image; and adjusting the height of the normalized image to an integer multiple of the height of the image block in the visual base model.

[0009] Furthermore, before performing clustering, the method also includes: performing dimensionality reduction processing on the global visual feature vectors of all images of the target heritage points to a preset dimension.

[0010] Furthermore, the scene cluster comprehensive score is a weighted sum of the logarithmic transformation value of the cluster size, the average similarity within the cluster, and the representativeness of the central image.

[0011] Furthermore, after extracting typical scenarios, the method also includes: For each typical scene, its center image is designated as the reference image, and the other images besides the center image are designated as images to be registered. Based on the reference image, each image to be registered undergoes standardized registration preprocessing, including: EXIF ​​orientation correction of the image to be registered, and converting the corrected image to RGB format; normalizing the width dimension while maintaining the aspect ratio of the converted RGB image; adjusting the height of the normalized image to an integer multiple of the height size required by the local feature matching model image; and generating a grayscale image of the resized image. The grayscale images corresponding to the reference image and each image to be registered are input into the offline trained local feature matching model. Feature matching is performed on each image to be registered and the reference image to obtain matching point pairs. Based on the matching point pairs, the homography matrix from each image to be registered to the reference image is estimated. The image to be registered is then transformed based on the homography matrix to obtain the transformed image and the corresponding number of matching point pairs, number of inliers, and inlier rate. If the number of matching point pairs, number of inliers, or inlier rate exceeds the preset threshold of the corresponding index, or if the scale of the transformed image exceeds the corresponding preset threshold, the image to be registered is deleted; otherwise, it is retained. For the images to be registered that have passed the initial registration quality screening, a unified canvas screening is further performed: the reference image is retained, and the other images to be registered that have passed the initial registration are sorted according to their corresponding inlier rate, number of inlier points, and / or number of matching point pairs; the images to be registered are added to the unified canvas with a preset maximum size constraint one by one in sequence, and the side length of the canvas after addition is calculated; if adding an image to be registered causes the side length of the canvas to exceed the preset maximum size constraint, the image to be registered is deleted, and the final set of effective registered images is obtained. Based on the corner coordinates of each effective registered image and their corresponding transformations in the homography matrix, the transformed corner coordinates of the effective registered image set in the reference image coordinate system are obtained. Based on the transformed corner coordinates, the unified canvas boundary range is determined, and a translation matrix is ​​constructed according to the unified canvas boundary range to project each effective registered image onto the unified canvas coordinate system as an image for transparent overlay. This yields a transparent overlay image of the current typical scene, which is used to represent the common visual frame formed by multiple users repeatedly shooting in the same typical scene.

[0012] Furthermore, the descending sorting is implemented as follows: first, sort by the inner point rate from high to low; if the inner point rates are the same, sort the same part by the number of inner points from high to low; if the number of inner points is also the same, sort the same part by the number of matching points from high to low.

[0013] Furthermore, the method also includes: generating a structural consistency heatmap based on a set of effectively registered images in a unified canvas coordinate system, implemented as follows: Under a unified canvas coordinate system, all valid registered images are converted into grayscale images, and the effective range of each valid registered image is determined based on a preset background threshold. At each pixel position on the unified canvas, the sum of grayscale values, the sum of squared grayscale values, and the number of effective coverages of all valid registered images are accumulated based on the effective range. Based on the sum of gray values ​​at each pixel location on the unified canvas and the number of effective coverages, calculate the average grayscale image in the unified canvas coordinate system; based on the sum of the squared gray values ​​at each pixel location, the number of effective coverages, and the average grayscale image, calculate the grayscale standard deviation at each pixel location, and normalize the grayscale standard deviations at all pixel locations in the unified canvas; subtract the normalized grayscale standard deviation from 1 to calculate the consistency map. Based on the average grayscale image, a gradient map is calculated; the consistency map is multiplied by the gradient map to obtain a structural consistency map; for pixel positions with an effective coverage count lower than a preset threshold, their heat value is set to zero, and the structural consistency map is spatially smoothed to obtain a structural consistency heatmap.

[0014] Furthermore, after obtaining the transparent overlay image, the method also includes: For each typical scenario, semantic segmentation is performed on each effective registration image participating in transparent overlay. The semantic segmentation result of each preset semantic category is upsampled to the size of the effective registration image and a corresponding binary mask is generated. Binary masks of the same preset semantic category are superimposed pixel by pixel in a unified canvas coordinate system to obtain a spatial frequency map of the preset semantic category; each pixel value in the frequency map represents the proportion of the preset semantic category appearing at that position; the spatial frequency map is displayed in pseudo-color and superimposed on a transparent overlay map to obtain a semantic frequency overlay map. Based on all valid registered images participating in transparent overlay, semantic element statistical indicators are calculated for the analysis of semantic element features, including: the proportion of valid registered images containing each preset semantic category element in all valid registered images participating in transparent overlay; and the proportion of the number of pixels of each preset semantic category element in all registered images.

[0015] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0016] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0017] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages: 1. This invention proposes an automatic extraction method for typical heritage scenes from user-generated images. First, it acquires a set of user-generated images of heritage sites and constructs a basic index table containing information such as image ID, width, height, and aspect ratio. The user-generated images undergo background cropping, size normalization preprocessing, and model input size adaptation. This transforms user-generated images from complex sources, with varying sizes or containing screenshot borders, into standardized image data that is traceable, computable, and suitable for visual model processing, reducing interference from invalid borders, size differences, and format inconsistencies in subsequent visual feature extraction. Second, it proposes a global visual feature representation and similarity calculation method for heritage images based on a visual basic model. Specifically, after pixel normalization of the preprocessed images, the data is input into the visual basic model to obtain a global visual feature vector for each image. This feature vector is then normalized, converting heritage images from different users, devices, and shooting conditions into a unified feature representation. Cosine similarity is calculated based on the normalized global visual feature vectors, which can stably characterize the similarity relationships between images, providing a foundation for heritage site scene clustering and typical scene selection. Furthermore, a typical scene identification method combining proximity relationships and density clustering within heritage sites is proposed. Specifically, by adaptively determining the minimum number of images to be included in a cluster and the minimum number of images used to determine the density core region based on the number of images at the heritage site, this method can adapt to situations where the number of images varies greatly among different heritage sites, avoiding the difficulty of clustering small-sample heritage sites or excessive fragmentation of large-sample heritage sites due to fixed parameters. By identifying non-noise scene clusters, relatively stable visual scenes formed by repeated shooting can be automatically discovered from a large number of user-generated images. Further, a typical scene quality evaluation and screening method based on cluster size, average intra-cluster similarity, and representativeness of the central image is proposed. Specifically, the image with the highest average cosine similarity to other member images within a cluster is determined as the central image, and a comprehensive score is calculated based on the logarithmic transformation value of the cluster size, average intra-cluster similarity, and representativeness of the central image. This method can both preserve the degree of public repeated shooting reflected by the number of images and avoid the excessive influence of large-scale image clusters on the screening results. Through effective cluster filtering and a high-quality cluster priority retention mechanism, typical heritage scenes with strong internal consistency, a clear central perspective, and high representativeness can be screened. Therefore, the method of this invention can form an automated process from user-generated image preprocessing, visual feature representation, scene clustering, and typical scene selection. This process can transform unstructured user-generated images into structured results with scenes, a central image, and member images other than the central image, reducing the workload of manually selecting typical viewpoints and judging scene features, improving the automation level of typical scene extraction for heritage sites, and is suitable for large-scale user-generated image analysis of different types of heritage sites.

[0018] 2. This invention further proposes a transparent overlay method for typical scene image clusters, based on geometric registration as the initial screening and a unified canvas as the secondary screening. Specifically, by using the center image as the reference image, a local feature matching model is used to obtain the matching point pairs between the image to be registered and the reference image, as well as the corresponding number of matching point pairs, number of inliers, and inlier ratio. Based on the matching point pairs, the homography matrix is ​​estimated, allowing images taken from different perspectives in the same typical scene to be projected into a unified space. Furthermore, by using the number of matching point pairs, number of inliers, inlier ratio, transformation scale, and unified canvas size constraints to screen effective registered images, overlay errors caused by incorrect matching, excessive perspective differences, and homography matrix anomalies can be reduced. The resulting transparent overlay image can intuitively express the common visual framework formed by repeated shooting of typical scenes.

[0019] 3. This invention further proposes a method for generating structural consistency heatmaps based on a unified canvas coordinate system. By statistically analyzing the sum of gray values, the sum of squared gray values, and the number of effective coverages of effectively registered images pixel by pixel in a unified canvas, a consistency map is calculated. This map is then combined with the gradient map of the average grayscale image to generate a structural consistency heatmap. This method can highlight recurring and stable common visual structures in multiple images, enhancing the stability and interpretability of extracting typical heritage scenes.

[0020] 4. This invention further proposes a semantic frequency mapping method based on a unified canvas coordinate system. By performing semantic segmentation on effectively registered images and superimposing binary masks of the same semantic category pixel by pixel in a unified canvas coordinate system, a spatial frequency map and a semantic frequency superposition map of semantic elements can be generated. This reveals the stable spatial distribution of different semantic elements in typical scenes, providing a quantitative basis for heritage scene feature recognition, public visual perception analysis, and heritage display optimization. Simultaneously, based on all effectively registered images participating in transparent overlay, semantic element statistical indicators are calculated for semantic element feature analysis, enhancing the interpretability of typical heritage scene extraction. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an automatic extraction method for typical heritage scenes based on user-generated images, provided as an embodiment of the present invention.

[0022] Figure 2 A flowchart illustrating the construction of the heritage point image metadata index table and the calculation of feature vectors provided in this embodiment of the invention.

[0023] Figure 3 The flowchart illustrates the nearest neighbor retrieval, scene clustering, and typical scene selection processes provided in this embodiment of the invention.

[0024] Figure 4This is a flowchart illustrating the generation process of transparent overlay images in a typical scene, as provided in an embodiment of the present invention.

[0025] Figure 5 A flowchart for generating a heatmap of structural consistency in a typical scenario provided in an embodiment of the present invention.

[0026] Figure 6 This is a flowchart illustrating the implementation process of the method for automatically extracting typical heritage scenes based on user-generated images, as provided in this embodiment of the invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0028] Example 1 An automatic extraction method for typical heritage scenes based on user-generated images, such as Figure 1 As shown, it includes: Obtain the user-generated image set of the target heritage point (each user-generated image contains the heritage point) and construct the corresponding basic index table. Each index includes the image ID of the user-generated image and its corresponding width, height and aspect ratio. Perform image background cropping and size normalization preprocessing on the user-generated image corresponding to each image ID. Each preprocessed image is pixel-normalized and then input into the visual base model to obtain the global visual feature vector of the preprocessed image, which is then normalized. The cosine similarity between images is calculated based on the normalized global visual feature vectors of each image. A density clustering algorithm is used to identify non-noise scene clusters based on the minimum number of images contained in a cluster adaptively preset for the target heritage point and the minimum number of images used to judge the density core region. Based on the cosine similarity between images, the cluster size (i.e., the number of images in the scene cluster), the average similarity within the cluster (i.e., the mean cosine similarity of images within the cluster), and the representativeness of the center image (i.e., the highest average cosine similarity value with other member images within the cluster) are calculated for each non-noise scene cluster to determine the comprehensive score of the scene cluster; where the center image is the image with the highest average cosine similarity with other member images within the cluster, and the representativeness of the center image is the highest average cosine similarity value. Scene clusters whose cluster size, average intra-cluster similarity, and central image representativeness all meet the preset thresholds of the corresponding indicators are recorded as valid clusters. A preset number of typical scene clusters are selected from the valid clusters: scene clusters whose average intra-cluster similarity and central image representativeness both reach the preset thresholds of the corresponding indicators are regarded as typical scene clusters (e.g., average intra-cluster similarity greater than or equal to 0.85, central image representativeness greater than or equal to 0.85). The remaining number of typical scene clusters are sorted according to the comprehensive score and the highest score is retained to complete the extraction of typical heritage scenes.

[0029] This embodiment presents an automatic method for extracting typical scenes from heritage sites. Traditionally, a typical scene refers to a relatively stable visual scene repeatedly expressed and documented by the public. In this invention, it specifically refers to a scene image cluster formed from multiple images with similar visual content, similar subjects, similar compositional relationships, or similar spatial perspectives within a user-generated image set of the same heritage site. Ultimately, this is expressed in this invention as a transparent overlay image constructed based on the scene image cluster.

[0030] When implementing the method of this embodiment, the following steps can be performed: (1) User-generated image standardization and visual feature extraction Please see Figure 2 First, user-generated images are organized into local folders based on heritage site names. Each heritage site's local folder contains a subfolder for valid generated images, used to store user-generated images that have undergone initial screening and are related to the heritage site, such as buildings (e.g., images irrelevant to the heritage scene, such as selfies, food, animals, and text screenshots, are removed using the CLIP image cleaning method).

[0031] The system iterates through the valid generated image subfolders corresponding to each heritage site, reading the local storage path, width, height, aspect ratio, and readability of each image. Furthermore, a unique image identifier (i.e., image ID) is constructed for each valid generated image, thereby generating a basic index table. This basic index table includes the image identifier, heritage site name, original width and height, original aspect ratio, and image readability. Simultaneously, an exception record table is generated for cases such as missing valid images or images that cannot be read.

[0032] Subsequently, each valid generated image undergoes preprocessing. Preferably, the preprocessing includes: EXIF ​​orientation correction of the user-generated image (to address the inconsistency between the actual pixel orientation and the display orientation of the photo, avoiding rotation errors when the image enters the model), and converting the corrected image to RGB format; estimating the background color based on the four corner pixels of the converted RGB image (using the four corner colors to estimate the background color, then identifying non-background areas with significant differences from the background color, and finally cropping based on the bounding rectangle of the non-background areas), identifying and cropping solid color borders; normalizing the width of the non-background area image while maintaining its aspect ratio; adjusting the height of the normalized image to an integer multiple of the height of the visual base model image block. For example, in a specific implementation, the short side of all images is normalized to 448, and the long side is scaled proportionally to ensure it does not exceed 896. The width and height are then aligned to integer multiples of 14. This is done to adapt to the batch structure of the DINOv2 ViT model (a visual base model), making the image size more suitable for model encoding. This preferred method acquires a set of user-generated images of heritage sites and constructs a basic index table containing information such as image ID, width, height, and aspect ratio. It then performs orientation correction, solid color border cropping, scale normalization, and model input size adaptation on the user-generated images. This transforms user-generated images with complex sources, varying sizes, and the presence of screenshot borders or abnormal orientations into standardized image data that is traceable, computable, and suitable for visual model processing. This reduces the interference of invalid borders, size differences, and inconsistent formats on subsequent visual feature extraction.

[0033] Through the above processing, preprocessed images are obtained and stored, and metadata for each preprocessed image is constructed, including the local storage path of each preprocessed image, the size before preprocessing, the size after preprocessing, and whether the border is cropped. Metadata is added to the basic index table to form an image metadata index table with preprocessing information.

[0034] Next, the preprocessed images are input into the visual base model to extract the global visual feature vector for each image. In a preferred embodiment, the visual base model can be the DINOv2 model; in other embodiments, CLIP, ViT, self-supervised visual representation models, or other visual coding models can also be used. Before inputting into the DINOv2 model, each valid image is pixel-normalized according to commonly used ImageNet mean and standard deviation (e.g., using commonly used ImageNet fixed mean and standard deviation: R channel: mean=0.485, std=0.229; G channel: mean=0.456, std=0.224; B channel: mean=0.406; std=0.225, because DINOv2 requires input images to be normalized). After extracting the global visual feature vectors of all images through the DINOv2 model, an image feature matrix is ​​constructed, and each row is normalized; each row represents one image. Here, normalization means dividing each image's own feature vector by its own length to allow features from different images to be compared. The L2 normalization method can be used, which involves dividing each vector by its own L2 norm to normalize the length of each image feature vector to 1.

[0035] (2) Neighbor retrieval, scene clustering and typical scene selection within heritage sites Please see Figure 3 The normalized image feature matrix and image metadata index table are read. For each heritage point, nearest neighbor retrieval and scene clustering are performed only among the image features in each row of the image feature matrix corresponding to that heritage point, in order to avoid image mixing between different heritage points that would cause distortion of the clustering results.

[0036] For each heritage point, the number of nearest neighbors k is adaptively determined based on the number of images. In a preferred embodiment, k is determined based on the square root of the number of images and is constrained by preset minimum and maximum values. Cosine similarity between images is calculated based on the normalized global feature vectors of the images, generating a nearest neighbor table for each heritage point. The nearest neighbor table includes image identifiers, image identifiers of k neighboring images, cosine similarity, nearest neighbor ranking, and mutual nearest neighbor labels. Two images are each other's k nearest neighbors, i.e., mutual nearest neighbors. When mutual nearest neighbor filtering is enabled, only the edges between two images that are each other's nearest neighbors are retained to improve the reliability of the nearest neighbor relationship. That is, each heritage point has a nearest neighbor table, and each image in the nearest neighbor table is configured with image identifiers of k neighboring images. The nearest neighbor ranking sorts the k neighboring images of each image according to the magnitude of the cosine similarity, represented by numbers 1, 2, 3, etc. The mutual nearest neighbor label is "TRUE" or "False".

[0037] Subsequently, to avoid instability caused by excessively high feature vector dimensionality, the global visual feature vector of the image to which each heritage point belongs can optionally be reduced in dimensionality. Density clustering parameters are adaptively set based on the number of images for that heritage point, including the minimum number of images a cluster must contain and the minimum number of samples used to determine the density core region. A density clustering algorithm is used to identify non-noise scene clusters, outputting the scene cluster label, cluster membership strength, and noise label for each image. In a preferred embodiment, the density clustering algorithm is HDBSCAN; in other embodiments, DBSCAN, OPTICS, or a community detection algorithm based on a nearest neighbor graph can also be used.

[0038] For each non-noise scene cluster, calculate the cluster size, average intra-cluster similarity, and representativeness of the center image. The center image is the image with the highest average cosine similarity to other member images within the cluster. Use the previously calculated cosine similarity to obtain the similarity between each member image and the center image, and determine the intra-cluster ranking of the member images.

[0039] The scene cluster comprehensive score is calculated based on cluster size, average intra-cluster similarity, and central image representativeness. Preferably, the scene cluster comprehensive score is a weighted sum of the logarithmic transformation value of the cluster size, the average intra-cluster similarity, and the central image representativeness. The logarithmic transformation value of the cluster size is used to avoid the influence of maxima on the calculation results; therefore, a logarithmic transformation value is adopted, allowing the cluster size to continue increasing with the number of images, but at a slower rate. Invalid clusters are filtered by setting thresholds for cluster size, average intra-cluster similarity, and central image representativeness. A threshold for the number of typical scene clusters is set to select typical scene clusters from the valid clusters. Scene clusters with both high average intra-cluster similarity and high central image representativeness are directly retained as typical scene clusters; for other valid clusters, the system selects the best ones based on their comprehensive scores.

[0040] Finally, a unique scene identifier is generated for each typical scene cluster, and a typical scene table and a typical scene member image table are output for each typical scene. The typical scene table includes the heritage site name, typical scene identifier, cluster size, average similarity within the cluster, representativeness of the central image, image identifier of the central image, and overall cluster score. The typical scene member image table includes the image identifier of the image belonging to the typical scene, heritage name, typical scene identifier information, and cluster order.

[0041] (3) Geometric registration and transparent overlay of typical scenes Please see Figure 4 Read each typical scene table and its corresponding typical scene member image table. For each typical scene, the aforementioned central image is selected as the reference image.

[0042] For each typical scenario, the center image is designated as the reference image, and the other images are designated as images to be registered. Based on the reference image, each image to be registered undergoes standardized registration preprocessing, including: EXIF ​​orientation correction (to address the inconsistency between the actual pixel orientation and the display orientation of the photo, and to avoid rotation errors when the image enters the model), and converting the corrected image to RGB format; normalizing the width dimension while maintaining the aspect ratio of the converted RGB image; adjusting the height of the normalized image to an integer multiple of the height requirement of the local feature matching model image; and generating a grayscale image of the resized image.

[0043] The reference image and the corresponding grayscale image of each image to be registered are input into an offline-trained local feature matching model. Feature matching is performed on each image to be registered and the reference image to obtain matching point pairs. In a preferred embodiment, the local feature matching model adopts LoFTR; in other embodiments, SuperGlue, LightGlue, SIFT, ORB or other local feature matching methods may also be used.

[0044] Based on matching point pairs, the RANSAC method can be used to estimate the homography matrix from the image to be registered to the reference image (the homography matrix transforms the pixel positions in the image to be registered to the coordinate system of the reference image). The image to be registered is then transformed based on the homography matrix to obtain the transformed image and the corresponding number of matching point pairs, number of inliers, and inlier ratio. After image transformation, registration quality screening is required. Specifically, based on thresholds set by the RANSAC method for the number of matching point pairs, number of inliers, and inlier ratio, if the number of matching point pairs, number of inliers, or inlier ratio exceeds the preset threshold for the corresponding indicator, or if the scale of the transformed image exceeds the corresponding preset threshold, it is filtered out; otherwise, it is retained.

[0045] For the retained images that have passed the initial registration quality screening, a unified canvas screening is further performed: the reference image is retained, and the other images that have passed the initial registration are sorted according to their corresponding inlier rate, number of inlier points, and / or number of matching point pairs (for example, sorted in descending order according to the priority of inlier rate, number of inlier points, and number of matching point pairs; specifically, sorted first by inlier rate from high to low; if the inlier rates are the same, sorted by the number of inlier points from high to low for the same part; if the number of inlier points is also the same, sorted by the number of matching points from high to low for the same part); the images to be registered are added to the unified canvas with a preset maximum size constraint one by one in sequence (for example, the preset canvas constraint is a maximum side length of 4000), and the side length of the canvas after addition is calculated; if adding an image to be registered causes the side length of the canvas to exceed the preset maximum size constraint, the image to be registered is deleted, and the final set of valid registered images is obtained.

[0046] After obtaining the final set of effective registered images, the corner coordinates of each effective registered image and their corresponding transformations in the homography matrix are used to obtain the transformed corner coordinates of the effective registered image set in the reference image coordinate system. Based on the transformed corner coordinates, the unified canvas boundary range is determined, and a translation matrix is ​​constructed according to the unified canvas boundary range to project each effective registered image onto the unified canvas coordinate system as an image for transparent overlay. This results in a transparent overlay image of the current typical scene, which is used to represent the common visual frame formed by multiple users repeatedly shooting in the same typical scene.

[0047] It's important to note that calculating the minimum and maximum coordinates of the boundary range and constructing the translation matrix are both for building a canvas that conforms to the set of effectively registered images. The reason for calculating the minimum and maximum coordinates of the boundary range is that after homography transformation, the image may not completely fall within the range of the reference image. If the size of the reference image is directly used as the canvas, these overflowing parts will be cropped. Therefore, it's necessary to calculate the corner coordinates (i.e., the minimum and maximum coordinates) of the set of effectively registered images in this typical scene to form the canvas boundary. The reason for constructing the translation matrix is ​​that image coordinates cannot be negative. After homography transformation, if the reference image coordinate system is still used as the reference, negative values ​​will exist. Therefore, a translation matrix must be used to translate the entire set of effective candidate images to ensure that there are no negative values. This process also changes the reference image coordinate system to a unified canvas coordinate system.

[0048] When constructing the overlay image, the images are further filtered: the reference image must be retained, and other images are selected according to the set threshold and the optimal selection within the cluster; the non-background area of ​​the selected image is superimposed on the white canvas with a preset transparency to generate a transparent overlay image; among them, the intra-cluster ranking of the member images is determined based on the similarity between each member image and the central image, with the images with higher similarity ranking first.

[0049] (4) Extraction of structural consistency heatmap and main hot spot region Please see Figure 5 It may include the following steps: (41) Read all valid registered images in the unified canvas coordinate system one by one and convert them into grayscale images. Determine the effective range of each valid registered image based on the preset background threshold. If the grayscale value of a pixel is lower than the preset background threshold, the pixel is determined to be a valid content area; if its grayscale value is not lower than the preset background threshold, it is determined to be an invalid background, so as to avoid blank backgrounds in the unified canvas from participating in subsequent statistics.

[0050] (42) At each pixel position of the unified canvas, perform pixel-by-pixel statistics on all valid registered images. For any pixel position, only if the position belongs to the valid range of the corresponding valid registered image, accumulate the gray value and gray square value of the pixel position, and increment the number of valid coverages by one to obtain the sum of gray values, the sum of gray square values ​​and the number of valid coverages of each pixel position in the unified canvas.

[0051] (43) Calculate the average grayscale image in a unified canvas coordinate system based on the sum of grayscale values ​​at each pixel location and the number of effective coverages. For any pixel location, the average grayscale value is the ratio of the sum of grayscale values ​​at that location to the number of effective coverages.

[0052] (44) Calculate the grayscale standard deviation for each pixel location based on the sum of the squared grayscale values, the number of effective coverages, and the average grayscale map. The grayscale standard deviation is used to characterize the degree of grayscale variation at the pixel location of all effectively registered images on the unified canvas; the smaller the grayscale standard deviation, the more stable the pixel location is across multiple images, and the higher the consistency. Normalize the grayscale standard deviation of all pixel locations in the unified canvas. Using the maximum value of the grayscale standard deviation in the current unified canvas as the normalization benchmark, transform the grayscale standard deviation of each pixel location to between 0 and 1 to obtain the normalized grayscale standard deviation map. Subtract the normalized grayscale standard deviation from 1 to obtain the consistency map. Pixel locations with smaller grayscale variations have higher consistency values.

[0053] (45) Calculate the gradient map based on the average grayscale image. The gradient map is calculated using the Sobel operator to measure the grayscale changes of the average grayscale image in the horizontal and vertical directions. For example, the weighted sum of grayscale values ​​of the pixels on the right (or below) is subtracted from the weighted sum of grayscale values ​​of the pixels on the left (or above) to determine the grayscale difference between the left and right (or top and bottom) sides at that position. Synthesize the gradient map: use the Euclidean norm to synthesize the gradient map, that is, take the square root after calculating the sum of squares, and synthesize the horizontal and vertical gradients into a gradient map.

[0054] (46) Multiply the corresponding pixels of the consistency map with the gradient map to obtain the structural consistency map. To avoid regions with only a single image being misjudged as stable structures, the structural consistency map is filtered based on the number of effective coverages. For pixel positions with an effective coverage count lower than a preset threshold, their heat value in the structural consistency map is set to zero. Thus, the structural consistency heatmap can suppress large flat areas such as the sky and pure walls, and highlight the stable and recurring building outlines, boundaries, components, and spatial structures in multiple images. The preset threshold for the number of effective coverages can be set to 2, that is, only pixel positions covered by at least two effectively registered images are retained. Spatial smoothing is performed on the structural consistency map to obtain the structural consistency heatmap. The spatial smoothing is used to reduce the influence of local noise and improve the continuity and interpretability of the heatmap region. The structural consistency map can explain the stability of the visual structure of the extracted typical scene.

[0055] (5) Semantic element frequency mapping and population statistics Read the transparent overlay image, the image table for transparent overlay, and the set of registered images corresponding to each typical scene. Perform semantic segmentation on each registered image to generate a binary mask with a preset semantic category. The preset semantic categories can include buildings, streets and squares, greenery, water bodies, sky, etc., and can also be expanded to walls, structures, roads, crowds, or other categories according to actual applications.

[0056] In a preferred embodiment, the semantic segmentation model uses SegFormer; in other embodiments, Mask2Former, OneFormer, DeepLab, SAM combined with a classifier, or other semantic segmentation models may also be used. The semantic segmentation results output by the model are upsampled to the registered image size, and a binary mask is generated for each preset semantic category.

[0057] For each typical scenario, semantic element statistics are calculated based on all successfully registered images. For a certain semantic category, the system counts the number of images containing that semantic element and divides it by the number of successfully registered images to obtain the frequency of occurrence of that semantic element; it counts the total number of pixels of that semantic element in all registered images and divides it by the total number of pixels in all registered images to obtain the area ratio of that semantic element; and it further calculates the average area ratio in a single image.

[0058] Simultaneously, based on the registered image subset used for visualization, binary masks of the same semantic category are cumulatively averaged pixel-by-pixel in a unified canvas coordinate system to obtain a spatial frequency map of that semantic category. Each pixel value in the frequency map represents the proportion of that semantic category appearing at that location. The semantic frequency map is displayed in pseudo-color and overlaid on a transparent overlay image to obtain a semantic frequency overlay map. Further calculations are made of metrics such as the average frequency, maximum frequency, proportion of regions above 0.5, and proportion of regions above 0.7 in the semantic frequency map.

[0059] In one alternative implementation, person detection can be further performed on the registered images. The person detection model can employ YOLO or other object detection models. The system statistically analyzes the number of people in each image, the presence of people, the presence of multiple people, the area ratio of person detection boxes, and the detection confidence, and summarizes the frequency of person occurrence, the frequency of multiple people occurrence, and the average area ratio of people at the scene level.

[0060] Through the above steps, typical scenes can be further transformed from a common framework at the visual structural level into semantic element stability, crowd activity status, and scene composition characteristics, providing a data foundation for heritage scene classification, scene evaluation, and impact mechanism analysis.

[0061] It should be noted that, for the specific implementation of this embodiment, please refer to [link / reference needed]. Figure 6 The design includes: acquiring user-generated images and social media information records organized by heritage site, and constructing an image list of heritage site-image-user record. Image orientation correction, border cropping, scale normalization, and visual model input size alignment are performed. Global features of the images are extracted using a visual basic model, and image neighbor similarity relationships are constructed according to heritage sites. Scene clusters are identified based on density clustering, and typical scenes are selected by combining cluster size, intra-cluster similarity, and representativeness of the central image. Using the central image of a typical scene as a reference image, feature matching, robust geometric registration, and unified canvas selection are performed on member images to generate transparent overlay images, structural consistency heatmaps, attention core images, and main hotspot region masks. Semantic segmentation is performed on the registered images to generate semantic element frequency maps and semantic statistical indicators. This method can automatically extract typical heritage scenes from scattered, heterogeneous, and perspective-varying user-generated images, and quantify their stable visual structure, compositional core, and semantic element distribution. Furthermore, the image metadata index table and neighbor edge table involved in this embodiment can be used for method call indexing and traceability.

[0062] Example 2 This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0063] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0064] The relevant technical solutions are the same as above, and will not be repeated here.

[0065] In summary, this invention proposes an automatic method for extracting typical scenes of heritage sites from user-generated images. This method forms a complete technical workflow from user-generated image preprocessing, visual feature representation, scene clustering, typical scene selection, geometric registration, transparent overlay, to structural consistency heatmap generation and semantic frequency mapping. This workflow can transform unstructured user-generated images into structured results with scenes, a central image, member images other than the central image, transparent overlay maps, structural consistency heatmaps, and semantic frequency maps. This reduces the workload of manually selecting typical viewpoints and judging scene features, improves the automation, repeatability, and interpretability of typical heritage scene extraction, and is applicable to large-scale user-generated image analysis of different types of heritage sites.

[0066] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically extracting typical heritage scenes based on user-generated images, characterized in that, include: Obtain the user-generated image set of the target heritage site and construct the corresponding basic index table. Each index includes the image ID of the user-generated image and its corresponding width, height and aspect ratio. Perform image background cropping and size normalization preprocessing on the user-generated image corresponding to each image ID. Each preprocessed image is pixel-normalized and then input into the visual base model to obtain the global visual feature vector of the preprocessed image, which is then normalized. The cosine similarity between images is calculated based on the normalized global visual feature vectors of each image. A density clustering algorithm is used to identify non-noise scene clusters based on the minimum number of images contained in a cluster adaptively preset for the target heritage point and the minimum number of images used to judge the density core region. Based on the cosine similarity between images, the cluster size, intra-cluster average similarity, and center image representativeness of each non-noise scene cluster are calculated to determine the overall score of the scene cluster. The center image is the image with the highest average cosine similarity to other member images in the cluster, and the center image representativeness is the highest average cosine similarity value. The cluster size is the number of images contained in the scene cluster, and the intra-cluster average similarity is the mean cosine similarity of the images within the cluster. Scene clusters that meet the preset thresholds for the corresponding indicators in terms of cluster size, average intra-cluster similarity, and representativeness of the central image are recorded as valid clusters. A preset number of typical scene clusters are selected from the valid clusters: scene clusters that meet the preset thresholds for the corresponding indicators in terms of average intra-cluster similarity and representativeness of the central image are selected as typical scene clusters. The remaining number of typical scene clusters are sorted according to the comprehensive score and selected with the highest score to complete the extraction of typical heritage scenes. The scene cluster comprehensive score is a weighted sum of the logarithmic transformation value of the cluster size, the average similarity within the cluster, and the representativeness of the central image.

2. The method for automatically extracting typical scenes of heritage sites as described in claim 1, characterized in that, The preprocessing includes: performing EXIF ​​orientation correction on the user-generated image and converting the corrected image to RGB format; estimating the background color based on the four corner pixels of the RGB converted image, identifying and cropping the solid color border; normalizing the width of the non-background area image while maintaining the aspect ratio of the cropped non-background area image; and adjusting the height of the normalized image to an integer multiple of the height of the image block in the visual base model.

3. The method for automatically extracting typical scenes of heritage sites as described in claim 1, characterized in that, Before performing clustering, the method also includes: performing dimensionality reduction processing on the global visual feature vectors of all images of the target heritage points to a preset dimension.

4. The method for automatically extracting typical scenes of heritage sites as described in claim 1, characterized in that, After extracting typical scenarios, the method also includes: For each typical scene, its center image is designated as the reference image, and the other images besides the center image are designated as images to be registered. Based on the reference image, each image to be registered undergoes standardized registration preprocessing, including: EXIF ​​orientation correction of the image to be registered, and converting the corrected image to RGB format; normalizing the width dimension while maintaining the aspect ratio of the converted RGB image; adjusting the height of the normalized image to an integer multiple of the height size required by the local feature matching model image; and generating a grayscale image of the resized image. The grayscale images corresponding to the reference image and each image to be registered are input into the offline trained local feature matching model. Feature matching is performed on each image to be registered and the reference image to obtain matching point pairs. Based on the matching point pairs, the homography matrix from each image to be registered to the reference image is estimated. The image to be registered is then transformed based on the homography matrix to obtain the transformed image and the corresponding number of matching point pairs, number of inliers, and inlier rate. If the number of matching point pairs, number of inliers, or inlier rate exceeds the preset threshold of the corresponding index, or if the scale of the transformed image exceeds the corresponding preset threshold, the image to be registered is deleted; otherwise, it is retained. For the images to be registered that have passed the initial registration quality screening, a unified canvas screening is further performed: the reference image is retained, and the other images to be registered that have passed the initial registration are sorted in descending order according to their corresponding inlier rate, number of inlier points, and / or number of matching point pairs; the images to be registered are added to the unified canvas with a preset maximum size constraint one by one in sequence, and the side length of the canvas after addition is calculated; if adding an image to be registered causes the side length of the canvas to exceed the preset maximum size constraint, the image to be registered is deleted, and the final set of effective registered images is obtained. Based on the corner coordinates of each effective registered image and their corresponding transformations in the homography matrix, the transformed corner coordinates of the effective registered image set in the reference image coordinate system are obtained. Based on the transformed corner coordinates, the unified canvas boundary range is determined, and a translation matrix is ​​constructed according to the unified canvas boundary range. Each effective registered image is projected onto the unified canvas coordinate system as an image for transparent overlay, thereby obtaining a transparent overlay image of the current typical scene, which is used to represent the common visual frame formed by multiple users repeatedly shooting in the same typical scene.

5. The method for automatically extracting typical scenes of heritage sites as described in claim 4, characterized in that, The descending sorting is implemented as follows: first sort by the internal point rate from high to low; if the internal point rates are the same, sort the same part by the number of internal points from high to low; if the number of internal points is also the same, sort the same part by the number of matching points from high to low.

6. The method for automatically extracting typical heritage scenes as described in claim 4, characterized in that, The method also includes: generating a structural consistency heatmap based on a set of effectively registered images in a unified canvas coordinate system, implemented as follows: Under a unified canvas coordinate system, all valid registered images are converted into grayscale images, and the effective range of each valid registered image is determined based on a preset background threshold. At each pixel position on the unified canvas, the sum of grayscale values, the sum of squared grayscale values, and the number of effective coverages of all valid registered images are accumulated based on the effective range. Based on the sum of gray values ​​at each pixel location on the unified canvas and the number of effective coverages, calculate the average grayscale image in the unified canvas coordinate system; based on the sum of the squared gray values ​​at each pixel location, the number of effective coverages, and the average grayscale image, calculate the grayscale standard deviation at each pixel location, and normalize the grayscale standard deviations at all pixel locations in the unified canvas; subtract the normalized grayscale standard deviation from 1 to calculate the consistency map. Based on the average grayscale image, a gradient map is calculated; the consistency map is multiplied by the gradient map to obtain a structural consistency map; for pixel positions with an effective coverage count lower than a preset threshold, their heat value is set to zero, and the structural consistency map is spatially smoothed to obtain a structural consistency heatmap.

7. The method for automatically extracting typical scenes of heritage sites as described in claim 4, characterized in that, After obtaining the transparent overlay image, the method also includes: For each typical scenario, semantic segmentation is performed on each effective registration image participating in transparent overlay. The semantic segmentation result of each preset semantic category is upsampled to the size of the effective registration image and a corresponding binary mask is generated. Binary masks of the same preset semantic category are superimposed pixel by pixel in a unified canvas coordinate system to obtain a spatial frequency map of the preset semantic category; each pixel value in the frequency map represents the proportion of the preset semantic category appearing at the corresponding pixel position; the spatial frequency map is displayed in pseudo-color and superimposed on a transparent overlay map to obtain a semantic frequency overlay map. Based on all valid registered images participating in transparent overlay, semantic element statistical indicators are calculated for the analysis of semantic element features, including: the proportion of valid registered images containing each preset semantic category element in all valid registered images participating in transparent overlay; and the proportion of the number of pixels of each preset semantic category element in all registered images.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium is located to perform the steps of the method as described in any one of claims 1 to 7.

Images