A label-based multi-modal cultural relic knowledge graph construction method, system and device

Abstract

The application provides a label-based multi-modal cultural relic knowledge graph construction method, system and device, relates to the technical field of cultural relic knowledge graph construction, and obtains multi-spectral image and texture image multi-modal data of a target rammed earth cultural relic, analyzes the multi-spectral image by relying on a pre-constructed standard reflection spectrum database of rammed earth materials, obtains material composition characteristics of each spatial region, and then generates material and topographic feature vectors by correlating corresponding texture images. Then, the vectors are mapped to initial features of graph network nodes, and updated aggregated features are obtained through iterative propagation and aggregation, and a composite label containing material attributes, disease types and spatial positions is generated accordingly, and finally, a target rammed earth cultural relic knowledge graph is constructed based on all composite labels, multi-modal data fusion of the rammed earth cultural relic can be realized, the composite label can be accurately generated and the knowledge graph can be constructed, and the application helps the systematic management and application of cultural relic information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cultural relic knowledge graph construction, and in particular to a label-based multi-modal cultural relic knowledge graph construction method, system and device. BACKGROUND

[0002] The cultural relic knowledge graph is an important tool for systematically integrating multi-dimensional information of cultural relics, and can provide data support for cultural relic protection, research and display, and has broad application prospects in the field of cultural heritage inheritance. It helps relevant personnel quickly master the core situation of cultural relics by associating information such as attributes, states and environments of cultural relics, and improves work efficiency and scientificity.

[0003] At present, the existing label-based multi-modal cultural relic knowledge graph construction method mostly obtains multi-type data such as images and texts of cultural relics, generates labels relying on simple data matching or basic feature extraction means, and then constructs a knowledge graph based on the labels. These methods have been applied in some cultural relic information integration scenarios, but the deep fusion of key features such as material and appearance of cultural relics has not been fully considered.

[0004] However, the current construction method has obvious deficiencies, and it is often difficult to accurately capture the specific attribute and state association of different spatial regions on the surface of cultural relics, resulting in that the generated label information is not comprehensive and accurate enough, and further affecting the practicability of the knowledge graph. At the same time, the analysis depth of multi-modal data of cultural relics is insufficient, and the potential association behind the data cannot be effectively mined. SUMMARY

[0005] The purpose of the present application is to provide a label-based multi-modal cultural relic knowledge graph construction method, system and device to solve the problem of insufficient construction accuracy and information integrity of the existing cultural relic knowledge graph.

[0006] To solve the above technical problems, in a first aspect, the present application provides a label-based multi-modal cultural relic knowledge graph construction method, comprising:

[0007] Obtaining multi-modal data of a target rammed earth cultural relic, the multi-modal data comprising multi-spectral image data covering the surface of the cultural relic, and texture image data corresponding to the texture of the surface of the cultural relic;

[0008] Based on a pre-constructed standard reflectance spectrum database of rammed earth materials, performing spectral analysis on the multi-spectral image data to obtain material composition features corresponding to each spatial region on the surface of the cultural relic;

[0009] Associating the material composition features corresponding to each spatial region with the corresponding texture image data to generate material feature vectors and appearance feature vectors corresponding to each spatial region;

[0010] mapping the material feature vector and the topography feature vector corresponding to each spatial region as initial features of corresponding nodes in a pre-constructed graph network, and performing iterative propagation and aggregation of node features on the graph network based on the initial features of the nodes and the connection relationship of edges to obtain updated aggregated features of each node;

[0011] generating a composite label for the spatial region corresponding to each node according to the updated aggregated features of the node, the composite label containing material attributes, disease types and spatial position information of the spatial region;

[0012] constructing a knowledge graph of the target rammed earth cultural relic according to all generated composite labels.

[0013] Optionally, the step of generating a composite label for the spatial region corresponding to each node according to the updated aggregated features of the node comprises:

[0014] inputting the updated aggregated features of each node in the graph network into a preset classification model to obtain material attribute categories and disease type categories corresponding to the node;

[0015] obtaining spatial position information represented by the node;

[0016] combining the material attribute categories, the disease type categories and the extracted spatial position information to generate a composite label corresponding to the node.

[0017] Optionally, the step of constructing a knowledge graph of the target rammed earth cultural relic according to all generated composite labels comprises:

[0018] creating or matching corresponding entities in a preset knowledge graph framework for material attributes, disease types and spatial position information parsed from the composite label;

[0019] establishing an initial attribute association between the entity representing the spatial position information and the entities representing the material attributes and the disease types;

[0020] integrating all created or matched entities and established initial attribute associations to complete the construction of the knowledge graph of the target rammed earth cultural relic.

[0021] Optionally, after the step of integrating all created or matched entities and established initial attribute associations to complete the construction of the knowledge graph of the target rammed earth cultural relic, the method further comprises:

[0022] In the knowledge graph, a group of spatial position entities is identified, wherein all spatial position entities in the group are adjacent to each other in the graph network and jointly point to the same material attribute entity or the same disease type entity through the initial attribute association;

[0023] For the spatial position entities in the group, a new aggregate entity is created, and the aggregate entity is integrated into the knowledge graph of the target rammed earth cultural relic. The aggregate entity is used to represent a continuous physical area composed of the spatial position entities in the group.

[0024] Optionally, before mapping the material feature vector and the topographic feature vector corresponding to each spatial region to the initial features of the corresponding nodes in the pre-constructed graph network, the method further comprises:

[0025] A graph network is constructed, the graph network comprising a plurality of nodes and edges connecting the nodes, wherein each node corresponds to a spatial region on the surface of the cultural relic, and the edges are constructed based on the adjacency relationship and feature similarity between the spatial regions.

[0026] Optionally, the step of performing spectral analysis on the multi-spectral image data based on the pre-constructed standard reflectance spectrum database of rammed earth materials to obtain the material composition feature corresponding to each spatial region on the surface of the cultural relic comprises:

[0027] For any spatial region in the multi-spectral image data, the spectral response sequence of the spatial region is extracted;

[0028] In the pre-constructed standard reflectance spectrum database of rammed earth materials, the standard reflectance spectrum with the highest similarity to the spectral response sequence of the spatial region is retrieved and determined;

[0029] If the highest similarity is higher than a preset similarity threshold, the rammed earth material corresponding to the standard reflectance spectrum is determined as the material composition feature of the spatial region.

[0030] Optionally, the step of associating the material composition feature corresponding to each spatial region with the corresponding texture image data to generate the material feature vector and the topographic feature vector corresponding to each spatial region comprises:

[0031] For each spatial region in the multi-spectral image data, the material composition feature corresponding to the spatial region is encoded to generate the material feature vector of the spatial region;

[0032] In the texture image data, a neighborhood centered on the spatial region is determined;

[0033] The texture value distribution in the neighborhood is counted to generate the topographic feature vector of the spatial region.

[0034] Optionally, the step of mapping the material feature vector and the topography feature vector corresponding to each spatial region as initial features of a corresponding node in a pre-constructed graph network, and performing iterative propagation and aggregation of node features on the graph network based on the initial features of the nodes and the connection relationship of the edges to obtain updated aggregated features of each node, comprises:

[0035] For each node in the graph network, aggregate the material feature vector and the topography feature vector of the spatial region corresponding to the node to generate the initial features of the node;

[0036] Based on the initial features of each node, repeatedly perform the following propagation and aggregation operations until a preset iteration termination condition is met:

[0037] For any target node in the graph network, obtain the current feature vectors of all adjacent nodes of the target node;

[0038] Aggregate the obtained current feature vectors of all adjacent nodes to generate a neighborhood aggregated feature;

[0039] Combine the current feature vector of the target node and the neighborhood aggregated feature to update the feature vector of the target node;

[0040] And the final updated node feature vector when the iteration termination condition is met is taken as the updated aggregated feature of the node.

[0041] In a second aspect, the present application provides a label-based multi-modal cultural relic knowledge graph construction system, comprising:

[0042] An acquisition module is configured to acquire multi-modal data of a target rammed earth cultural relic, wherein the multi-modal data comprises multi-spectral image data covering the surface of the cultural relic and texture image data corresponding to the texture of the surface of the cultural relic;

[0043] A determination module is configured to perform spectral analysis on the multi-spectral image data based on a pre-constructed standard reflectance spectrum database of rammed earth materials to obtain material composition features corresponding to each spatial region on the surface of the cultural relic;

[0044] A generation module is configured to associate the material composition features corresponding to each spatial region with the corresponding texture image data to generate a material feature vector and a topography feature vector corresponding to each spatial region;

[0045] a processing module, configured to map the material feature vector and the topography feature vector corresponding to each spatial region as initial features of corresponding nodes in a pre-constructed graph network, and perform iterative propagation and aggregation of node features on the graph network based on the initial features of the nodes and connection relationships of edges to obtain updated aggregated features of each node;

[0046] The generation module is further configured to generate a composite label for the spatial region corresponding to each node according to the updated aggregated features of the node, where the composite label contains material attributes, disease types, and spatial position information of the spatial region.

[0047] The construction module is configured to construct a knowledge graph of the target rammed earth cultural relic according to all generated composite labels.

[0048] In a third aspect, the present application provides an electronic device, comprising:

[0049] a memory configured to store a computer program;

[0050] a processor configured to implement the steps of the label-based multi-modal cultural relic knowledge graph construction method according to the first aspect when the computer program is executed.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can implement the steps of the label-based multi-modal cultural relic knowledge graph construction method according to the first aspect when the computer program is executed by a processor.

[0052] The label-based multi-modal cultural relic knowledge graph construction method provided by the present application can comprehensively collect information related to the surface of a cultural relic by obtaining multi-spectral image and texture image multi-modal data of a target rammed earth cultural relic, and provide data support for subsequent analysis. The material composition features of each spatial region of the cultural relic can be accurately obtained by analyzing the multi-spectral image data based on a pre-constructed standard reflectance spectrum database of rammed earth materials. The material and topography feature vectors can be generated by associating the material composition features with the texture image data, so as to integrate key feature information of the cultural relic and facilitate subsequent processing. The features can be mined and the feature expression can be optimized by mapping the feature vectors as initial features of graph network nodes and performing iterative propagation and aggregation. The key information of each region of the cultural relic can be accurately labeled by generating a composite label containing material attributes, disease types, and spatial positions according to the updated aggregated features. The cultural relic information can be systematically integrated to realize ordered management of information by constructing a knowledge graph based on all composite labels.

[0053] Further, for each node in the graph network, the updated aggregated features are input into a preset classification model to parse the material attribute category and the disease type category, and then the spatial position information represented by the node is obtained, and finally the three types of information are combined to generate a composite label. This step can further improve the accuracy and standardization of composite label generation, ensure the completeness and actual situation of the label information, and provide more reliable label support for high-quality construction of the knowledge graph. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 A flowchart of a label-based multi-modal cultural relic knowledge graph construction method provided by an embodiment of the present application;

[0056] Figure 2 A feature mapping diagram of a label-based multi-modal cultural relic knowledge graph construction method provided by an embodiment of the present application;

[0057] Figure 3 A graph network iteration propagation aggregation diagram of a label-based multi-modal cultural relic knowledge graph construction method provided by an embodiment of the present application;

[0058] Figure 4 A knowledge graph construction diagram of a label-based multi-modal cultural relic knowledge graph construction method provided by an embodiment of the present application;

[0059] Figure 5 A structure diagram of a label-based multi-modal cultural relic knowledge graph construction system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In the field of cultural relic knowledge graph construction, the existing label-based multi-modal construction method has obvious limitations: only simple data matching or basic feature extraction is used to generate labels, key information such as cultural relic material and morphology is not deeply fused, it is difficult to accurately capture the attributes and state correlations of different spatial regions on the surface of cultural relics, resulting in one-sided and inaccurate label information, and the potential relationship behind multi-modal data cannot be mined, ultimately causing insufficient accuracy and information integrity of the knowledge graph construction, and it is difficult to meet the actual needs of cultural relic protection and research.

[0061] To solve the problem, the application provides a label-based multi-modal cultural relic knowledge graph construction method, the core of which is to generate accurate labels through multi-dimensional data processing and deep fusion. Specifically, first, multi-spectral image and texture image data of cultural relics are obtained, material composition characteristics are analyzed with the help of a standard reflectance spectrum database, and then material and topographic feature vectors are generated by associating texture data, and the feature expression is optimized through graph network iteration and aggregation, and finally a composite label containing material attributes, disease types and spatial positions is generated, and a knowledge graph is constructed accordingly. This method realizes the deep fusion of key features of cultural relics and the mining of potential data associations, accurately compensates for the defects of existing methods such as incomplete labels and insufficient information integration, effectively improves the construction quality of cultural relic knowledge graph, and provides more reliable information support for cultural relic-related work.

[0062] To make the person skilled in the art better understand the scheme of the application, the application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0063] The core of the application is to provide a label-based multi-modal cultural relic knowledge graph construction method, and a flowchart of a specific embodiment of the method is shown in Figure 1 The method comprises the following steps.

[0064] S101, obtaining multi-modal data of a target rammed earth cultural relic.

[0065] The target rammed earth cultural relic is mainly for non-movable rammed earth relics, such as the Western Xia Mausoleum relics. The multi-modal data includes multi-spectral image data covering the surface of the cultural relic, and texture image data corresponding to the surface texture of the cultural relic. The multi-spectral image data includes images reflecting the reflection ability of different wavebands of the surface of the cultural relic, which can present the characteristics of different materials. The texture image data is used to clearly present the crack, peeling trace, weathering and other detail states of the surface of the cultural relic.

[0066] In a specific embodiment, a multi-spectral camera-equipped unmanned aerial vehicle is used to take a full-range aerial photograph of the rammed earth cultural relic, capture reflection information of different wavebands of the surface of the cultural relic, and form multi-spectral image data; at the same time, a high-definition camera-equipped unmanned aerial vehicle is used to take dense photographs of the surface of the cultural relic at a centimeter to millimeter level resolution by using close-range photogrammetry technology, and planning the flight route in advance, controlling the appropriate flight height and shooting angle, and obtaining texture image data that can accurately restore the surface texture.

[0067] S102, based on the pre-constructed standard reflection spectrum database of rammed earth materials, performing spectral analysis on the multi-spectral image data to obtain material composition characteristics corresponding to each spatial region on the surface of the cultural relic.

[0068] The standard reflection spectrum database of rammed earth materials is a collection of reflection characteristic data of different types of rammed earth materials at each wave band of the multi-spectrum, and includes standard reflection data of various common rammed earth materials such as healthy dry rammed earth, salt-enriched rammed earth, high water content rammed earth, and biological membrane covered rammed earth. These data can be used as a reference basis for material composition identification. The material composition characteristics are used to determine the material attribute state of each spatial region on the surface of the cultural relic, such as salt enrichment state, water content distribution, weathering degree, and the like.

[0069] Optionally, step S102 can specifically include the following steps:

[0070] S1021, for any spatial region in the multi-spectral image data, extracting the spectral response sequence of the spatial region.

[0071] The spectral response sequence refers to an ordered data combination of reflection values of a specific spatial region at different wave bands of the multi-spectral image, which can reflect the reflection characteristics of the material of the region to light at different wave bands. The spatial region can be a pixel in the multi-spectral image data, or a pixel set constructed by multiple pixel points, which can be set according to requirements.

[0072] In a specific embodiment, the multi-spectral image is first divided into a plurality of independent spatial regions according to a predetermined spatial resolution, and each spatial region corresponds to a pixel or a pixel block in the image. Then, the reflection intensity values of each spatial region at each wave band of the multi-spectral image, such as the visible light wave band and the near-infrared wave band, are extracted by image processing technology. Arranging these values in order of wave band, the spectral response sequence of the spatial region is obtained.

[0073] For example, continuing the above example, the multi-spectral image of a certain rammed earth cultural relic is divided into 10,000 spatial regions, and one of the spatial regions located in the middle of the cultural relic is selected. The reflection values of the spatial region at four wave bands are 0.2, 0.4, 0.6, and 0.8, respectively. After arranging the reflection values in order of wave band, the spectral response sequence of the region is [0.2, 0.4, 0.6, 0.8]. The above example is only an example of the present application, and in actual application, the division of the spatial region can be adjusted according to the image resolution and analysis requirements, which is not limited in the present application.

[0074] S1022, in the pre-constructed standard reflection spectrum database of rammed earth materials, searching and determining the standard reflection spectrum with the highest similarity to the spectral response sequence of the spatial region.

[0075] The standard reflection spectrum is a typical spectral response sequence of each known rammed earth material stored in a standard reflection spectrum database of rammed earth materials, and is a reference benchmark for judging the material composition of an unknown spatial region.

[0076] In a specific embodiment, the cosine similarity algorithm is first used to calculate the similarity between the spectral response sequence of the spatial region to be identified and each standard reflection spectrum in the database. The core of the cosine similarity algorithm is to measure the similarity between two vectors by calculating the cosine value of the included angle between them. Formula (1) is as follows:

[0077] (1)

[0078] In formula (1), is the cosine similarity value, which ranges from 0 to 1. The closer the value is to 1, the higher the similarity is; is the number of spectral bands; is the reflectance value of the i-th band in the spectral response sequence of the spatial region to be identified; is the reflectance value of the i-th band in a standard reflection spectrum in the database. After the calculation is completed, all similarity values are compared, and the standard reflection spectrum corresponding to the largest similarity value is selected. This spectrum is the standard reflection spectrum with the highest similarity to the spectral response sequence of the spatial region to be identified. In another specific embodiment, the Euclidean distance algorithm can also be used to calculate the similarity. The smaller the Euclidean distance, the higher the similarity. Formula (2) is as follows:

[0079]

[0080] (2)

[0081] In formula (2), is the Euclidean distance;

[0082] is the number of spectral bands; is the reflectance value of the i-th band in the spectral response sequence of the spatial region to be identified; is the reflectance value of the i-th band in a standard reflection spectrum in the database.

[0083] ​​​​​In practical applications, continuing the example in the above steps, the spectral response sequence of the spatial region is [0.2, 0.4, 0.6, 0.8], and the database contains the standard reflectance spectrum of healthy dry rammed earth, salt-enriched rammed earth, and high water content rammed earth, which are [0.22, 0.42, 0.62, 0.82], [0.5, 0.7, 0.9, 0.95], and [0.7, 0.5, 0.3, 0.1], respectively.

[0084] The similarity is calculated by formula (1):

[0085] The similarity of the spectral response sequence of the spatial region and the standard spectrum of healthy dry rammed earth is:

[0086] ;

[0087] The similarity of the spectral response sequence of the spatial region and the standard spectrum of salt-enriched rammed earth is:

[0088] ;

[0089] The similarity of the spectral response sequence of the spatial region and the standard spectrum of high water content rammed earth is:

[0090] .

[0091] Comparing the three similarity values 0.982, 1.008, and 0.597, the highest similarity is 1.008, and if it is greater than 1, the value is 1, and the corresponding standard reflectance spectrum is the standard reflectance spectrum of salt-enriched rammed earth. The above example is only one example of the present application, and other similarity calculation algorithms can be selected according to the data characteristics in practical applications, which are not limited by the present application.

[0092] S1023, if the highest similarity is higher than the preset similarity threshold, the standard reflectance spectrum corresponding to the rammed earth material is determined as the material composition feature of the spatial region.

[0093] The preset similarity threshold is a critical value for judging whether the spectrum of the spatial region to be identified matches the standard reflectance spectrum, which is set according to the actual recognition accuracy requirement.

[0094] In the embodiments of the present application, first, according to a large number of rammed earth material spectrum matching experimental results, combined with the requirements of material identification accuracy in the field of cultural relic protection, a preset similarity threshold is set to 0.95. Then, the highest similarity obtained in S1022 is compared with the threshold. If the highest similarity is greater than 0.95, it is determined that the material of the spatial region is consistent with the rammed earth material corresponding to the standard reflectance spectrum, and the rammed earth material corresponding to the standard reflectance spectrum is determined as the material component feature of the spatial region. If the highest similarity is less than or equal to 0.95, it is determined that the matching fails, and the spectral data needs to be rechecked or the standard reflectance spectrum in the database needs to be supplemented.

[0095] Through the standardized spectral database and the accurate similarity calculation algorithm, the present application realizes the rapid and accurate identification of the material components on the surface of the rammed earth cultural relics, and avoids the subjectivity and limitations of traditional manual detection.

[0096] S103, associating the material component feature corresponding to each spatial region with the corresponding texture image data to generate a material feature vector and a topographic feature vector corresponding to each spatial region.

[0097] In this step, the material feature vector is an ordered data combination formed after the material component feature is digitally coded, which is used to quantitatively represent the material properties of the spatial region; the topographic feature vector is a data vector generated based on the texture distribution information of the spatial region and its periphery in the texture image, which is used to reflect the surface structure state of the spatial region; the two together constitute the comprehensive feature description of the spatial region, realizing the association and integration of material properties and surface topography information.

[0098] Optionally, step S103 can specifically include the following steps:

[0099] S1031, for each spatial region in the multispectral image data, coding the material component feature corresponding to the spatial region to generate a material feature vector of the spatial region.

[0100] Wherein, the coding is the process of converting the non-digital material component feature into a digital form that can be recognized and processed by a computer, and the correspondence between the material component feature and the digital vector is established through a preset coding rule.

[0101] In a specific implementation, first, all possible material component feature types are sorted out, such as healthy dry rammed earth, salt-enriched rammed earth, high water content rammed earth, and biological membrane covered rammed earth, and a unique digital identifier is assigned to each feature type. Then, a one-hot coding method is used to convert the material component feature of each spatial region into a corresponding binary vector, the vector length is equal to the total number of material component feature types, only the position corresponding to the feature type is 1, and the remaining positions are 0, to generate the material feature vector of the spatial region.

[0102] In another specific embodiment, a numerical coding method can also be used, and different values are assigned according to the relevant properties of the material composition, such as water content and salt content, and these values are arranged in a predetermined order to form the material feature vector.

[0103] For example, assuming that there are four types of material composition features, namely, healthy dry rammed earth No. 1, salt-enriched rammed earth No. 2, high water content rammed earth No. 3, and biological membrane covered rammed earth No. 4. Using the one-hot coding rule, if the material composition feature of a space region is salt-enriched rammed earth, the corresponding material feature vector is [0, 1, 0, 0]; if the material composition feature is healthy dry rammed earth, the material feature vector is [1, 0, 0, 0]. The above example is only an example of the present application, and in actual application, the coding method can be adjusted according to the number and type of material composition features, which is not limited in the present application.

[0104] S1032, in the texture image data, a neighborhood centered on the space region is determined.

[0105] The neighborhood refers to a certain range of surrounding regions in the texture image centered on the target space region, which is used to comprehensively consider the texture information of the target region and its surrounding area, and more comprehensively reflect the surface topographic features of the target region.

[0106] In a specific embodiment, the size parameters of the neighborhood are set according to the resolution of the texture image and the size of the space region. If the space region corresponds to a pixel in the texture image, the neighborhood can be set as a 3x3 or 5x5 pixel matrix, i.e., the target pixel is centered, and the adjacent pixels in the up, down, left, right and diagonal directions are included. The position of the target space region in the texture image is determined by coordinate positioning, and then the corresponding range centered on the position is selected according to the set neighborhood size, which is the neighborhood of the target space region.

[0107] For example, continuing the previous example, assuming that the space region is a pixel point, and the pixel coordinates corresponding to a certain space region in the texture image are (50, 50), and the neighborhood size is set as a 3x3 pixel matrix. With (50, 50) as the center, 9 pixels with coordinates (49, 49), (49, 50), (49, 51), (50, 49), (50, 50), (50, 51), (51, 49), (51, 50), and (51, 51) are selected to form the neighborhood of the space region. If the space region is a region composed of multiple pixel points, the neighborhood of the space region can also be determined in the above manner.

[0108] S1033, the texture value distribution in the neighborhood is counted to generate the topographic feature vector of the space region.

[0109] The texture value distribution refers to the value and distribution of texture parameters, such as a gray value, a contrast, a roughness, and the like, of each pixel in the neighborhood, and is a core basis for generating the topographic feature vector.

[0110] In a specific embodiment, first, the gray value of each pixel in the neighborhood is extracted as a texture value, and then the statistical features of the gray values in the neighborhood are calculated, including an average value, a variance, a maximum value, a minimum value, a median, and the like. The average value is used to reflect the overall light and dark degree of the texture in the neighborhood, and is obtained by summing all the pixel gray values and dividing by the total number of pixels. The variance is used to reflect the dispersion degree of the gray values in the neighborhood, that is, the roughness of the texture, and is obtained by calculating the square of the difference between each pixel gray value and the average value and then averaging. These statistical features are arranged in a predetermined order to form the topographic feature vector of the spatial region, achieving quantitative description of the surface topography of the spatial region.

[0111] For example, continuing the example in the above steps, assuming that the gray values of the 9 pixels in the 3x3 neighborhood of a certain spatial region are [120, 125, 122, 123, 121, 124, 126, 123, 122]. Then, the average value is calculated: the sum of the 9 gray values is 1096, and then divided by the total number of pixels 9, i.e. 1096 ÷ 9 ≈ 121.78.

[0112] Then the variance is calculated: first, the square of the difference between each gray value and the average value 121.78 is calculated, which is , , , , , , , , the sum of these squared differences is 39.968, and then divided by the total number of pixels 9, i.e. .

[0113] At the same time, the maximum value of the gray values in the neighborhood is determined to be 126, the minimum value is 120, and the median is 123. In the order of "average value, variance, maximum value, minimum value, median", the topographic feature vector of the spatial region is obtained as [121.78, 4.44, 126, 120, 123]. The above example is only an example of the present application, and other texture parameters or statistical features can be selected to generate the topographic feature vector in actual application, which is not limited by the present application.

[0114] The application realizes effective association of material component features and texture information through coding technology and neighborhood statistical analysis, the generated feature vector can comprehensively and quantitatively describe the material and topography attributes of the spatial region, overcomes the limitation of traditional single feature description, and improves the integrity and usability of feature information.

[0115] In S104, the material feature vector and the topography feature vector corresponding to each spatial region are mapped into initial features of corresponding nodes in a pre-constructed graph network, and based on the initial features of the nodes and the connection relationship of the edges, iterative propagation and aggregation of node features are performed on the graph network to obtain updated aggregated features of each node.

[0116] In this step, the graph network is a structured network model composed of nodes and edges, the nodes correspond to spatial regions on the surface of cultural relics, and the edges are used to represent the association relationship between the nodes, such as geographical adjacency or feature similarity; the initial feature is the basic feature data of the node, which is generated by aggregating the material feature vector and the topography feature vector of the corresponding spatial region; the iterative propagation and aggregation means that in the graph network, the nodes constantly receive feature information of adjacent nodes through the connection relationship of the edges and update the information, and gradually improve their own features; the aggregated feature is the feature vector finally formed by the nodes after multiple iterative propagation and aggregation, which can comprehensively reflect the feature information of the node itself and the surrounding associated nodes. Specifically, as shown in Figure 2 The spatial regions 1, 2, 3, and 4 are obtained by pre-dividing the surface of the cultural relic, and it should be noted that the division results of the above four spatial regions are only examples, and actually N spatial regions can be divided. The material feature vector and the topography feature vector corresponding to each spatial region can be determined through the calculation process of the above steps 101-103, such as the material feature vector A1 and the topography feature vector B1 corresponding to the spatial region 1.

[0117] The spatial region is mapped into a node in the graph network, and the multiple vectors corresponding to the spatial region are mapped into the initial features of the node. As shown in Figure 2 For example, the spatial region 1 is mapped into the node 1 in the graph network, and the material feature vector A1 and the topography feature vector B1 corresponding to the spatial region 1 are mapped into the initial features C1 of the node 1.

[0118] Further, based on the initial features of the nodes and the connection relationship of the edges, iterative propagation and aggregation of node features are performed on the graph network to obtain updated aggregated features of each node. As shown in Figure 3 The initial feature C1 of the node 1 is iteratively propagated and aggregated to obtain the aggregated feature, wherein the aggregated feature is the aggregation result of the initial feature C1 and the current feature vector of all adjacent nodes. The following is the specific implementation process of step S104:

[0119] Optionally, step S104 can specifically include the following steps:

[0120] S1041, for each node in the graph network, aggregate the material feature vector and the topography feature vector of the space region corresponding to the node to generate an initial feature of the node.

[0121] The aggregation is a process of combining the two types of feature data of different dimensions, i.e., the material feature vector and the topography feature vector, to form comprehensive feature data of a uniform dimension, for comprehensively representing the space region attribute corresponding to the node. In a specific implementation, the vector splicing manner is used for aggregation, and the material feature vector and the topography feature vector corresponding to each node are directly connected in sequence to form a longer vector as the initial feature of the node. The vector splicing does not need to change the numerical values of the original feature data, and only needs to expand the dimension to realize feature fusion, which is simple to operate and can completely retain the original information of the two types of features.

[0122] In another specific implementation, the weighted summation manner can also be used for aggregation, and weights are assigned according to the importance of the material feature and the topography feature, and the numerical values of the corresponding dimensions of the two types of vectors are weighted and added to obtain the initial feature vector.

[0123] For example, continuing the previous example, the material feature vector corresponding to a node is [0, 1, 0, 0], and the topography feature vector is [121.78, 4.44, 126, 120, 123]. The vector splicing manner is used for aggregation, and the two vectors are connected in the order of “material feature vector in front and topography feature vector in back” to obtain the initial feature of the node as [0, 1, 0, 0, 121.78, 4.44, 126, 120, 123].

[0124] S1042, based on the initial feature of each node, repeatedly perform the following propagation and aggregation operations until a preset iteration termination condition is met.

[0125] The iteration termination condition is a basis for judging whether the iteration propagation and aggregation operation is stopped, for controlling the number of iterations or ensuring that the feature update reaches a stable state, and common ones include a preset fixed number of iterations, a feature change amount of the node in adjacent two iterations being less than a set threshold, etc.

[0126] Specifically, the preset fixed number of iterations is used as the termination condition, and according to the number of space regions on the surface of cultural relics and the complexity of association, the number of iterations is set to 3-5 times. When the number of iterations reaches the preset value, the propagation and aggregation operation is stopped, and the feature vector of the node at this time is the updated aggregated feature.

[0127] In another specific embodiment, the termination condition is that the change in node feature is less than a threshold value. The average difference between the feature vector of all nodes after each iteration and the feature vector of the last iteration is calculated. When the average difference is less than a preset threshold value, it indicates that the feature update has stabilized, and the iteration operation is stopped.

[0128] For example, for a graph network containing 1000 nodes, considering that the node association relationship is relatively simple, the preset iteration termination condition is to iterate 3 times. After completing 3 rounds of propagation and aggregation operations, the iteration is stopped and the feature vector of each node at this time is retained. The above example is only an example of the present application, and in actual application, the iteration termination condition can also be adjusted according to the actual scene, which is not limited by the present application.

[0129] S1043, for any target node in the graph network, the current feature vector of all adjacent nodes of the target node is obtained.

[0130] Among them, the target node refers to the node currently being updated; the adjacent node refers to the node directly connected to the target node through the edge, and the current feature vector refers to the feature vector of the adjacent node that has not been updated in this iteration, or the initial feature vector in the first iteration.

[0131] Specifically, the adjacency matrix of the graph network is constructed in advance, and the value of the element in the adjacency matrix is used to identify whether there is an adjacent relationship between the nodes. If the value is 1, it means that the corresponding two nodes are adjacent nodes, and if the value is 0, it means that they are not adjacent nodes. For any target node, the elements in the corresponding row or column of the adjacency matrix are queried to filter out the nodes corresponding to the positions with a value of 1, which are all adjacent nodes of the target node, and then the current feature vectors of these adjacent nodes are extracted from the feature storage container.

[0132] For example, continuing the previous example, the target node is set to node 2, and its adjacent nodes in the graph network are node 1, node 3, and node 4. After confirming the adjacent relationship by querying the adjacency matrix, the current feature vectors of node 1, node 3, and node 4 in this iteration are extracted from the feature storage container, which are [1, 0, 0, 0, 120.50, 3.20, 125, 118, 122], [0, 0, 1, 0, 122.30, 5.10, 127, 121, 124], and [0, 0, 0, 1, 121.10, 4.00, 124, 119, 121], respectively. The above example is only an example of the present application, and in actual application, the adjacent nodes can also be queried and the feature vectors can also be obtained by other means, which is not limited by the present application.

[0133] S1044, the current feature vectors of all adjacent nodes obtained are aggregated to generate a neighborhood aggregated feature.

[0134] The neighborhood aggregation feature is a feature vector obtained by comprehensively processing current feature vectors of all adjacent nodes, and is used to collectively reflect the overall feature state of the neighborhood of the target node.

[0135] In a specific embodiment, the neighborhood aggregation feature is generated in a mean aggregation manner, the values of corresponding dimensions of the current feature vectors of all adjacent nodes are summed respectively, and then divided by the number of adjacent nodes to obtain the average value of each dimension. The average values are arranged in order to form the neighborhood aggregation feature. The mean aggregation can balance the feature influence of all adjacent nodes, and is simple and stable to calculate.

[0136] For example, continuing the example in the above steps, the current feature vectors of the 3 adjacent nodes of the target node 2 are node 1: [1, 0, 0, 0, 120.50, 3.20, 125, 118, 122], node 3: [0, 0, 1, 0, 122.30, 5.10, 127, 121, 124], and node 4: [0, 0, 0, 1, 121.10, 4.00, 124, 119, 121].

[0137] The average values of each dimension are calculated: the 1st dimension: (1+0+0) ÷ 3≈0.33; the 2nd dimension: (0+0+0) ÷ 3=0; the 3rd dimension: (0+1+0) ÷ 3≈0.33; the 4th dimension: (0+0+1) ÷ 3≈0.33; the 5th dimension: (120.50+122.30+121.10) ÷ 3=363.90 ÷ 3=121.30; the 6th dimension: (3.20+5.10+4.00) ÷ 3=12.30 ÷ 3=4.10; the 7th dimension: (125+127+124) ÷ 3=376 ÷ 3≈125.33; the 8th dimension: (118+121+119) ÷ 3=358 ÷ 3≈119.33; and the 9th dimension: (122+124+121) ÷ 3=367 ÷ 3≈122.33.

[0138] Then, the average values of each dimension are arranged in order to obtain the neighborhood aggregation feature [0.33, 0, 0.33, 0.33, 121.30, 4.10, 125.33, 119.33, 122.33]. The above example is only an example of the present application, and in actual application, the neighborhood aggregation feature can also be generated in other manners such as summation aggregation and maximum value aggregation, which are not limited in the present application.

[0139] S1045, the current feature vector of the target node is combined with the neighborhood aggregation feature to update the feature vector of the target node.

[0140] In the combination, the current feature vector of the target node is fused with the neighborhood aggregated feature to generate a new feature vector that is more comprehensive and can reflect the actual state of the node, which is the core step of updating the node feature.

[0141] In a specific embodiment, the combination is performed by weighted fusion, and weights are set for the current feature vector of the target node and the neighborhood aggregated feature, and the sum of the weights is 1. The values of the corresponding dimensions of the two types of feature vectors are multiplied by the respective weights and then added to obtain the updated feature vector of the target node. By adjusting the weights, the influence of the node's own feature and the neighborhood feature on the update result can be controlled. Generally, the weight of the node's own feature is set to be higher than that of the neighborhood aggregated feature to ensure the dominant position of the node's own attribute.

[0142] The fusion formula (3) is as follows:

[0143] (3)

[0144] In formula (3), is the updated feature vector of the target node; is the weight of the current feature vector of the target node, and the value range is between 0 and 1; is the current feature vector of the target node; is the weight of the neighborhood aggregated feature; is the neighborhood aggregated feature.

[0145] For example, continuing the previous example, the current feature vector of the target node 2 is is , the neighborhood aggregated feature is , and the weight is set to be .

[0146] According to formula (3), the updated values of each dimension are calculated: the first dimension: 0*0.7+0.33*0.3≈0+0.10=0.10; the second dimension: 1*0.7+0*0.3=0.7+0=0.7; the third dimension: 0*0.7+0.33*0.3≈0+0.10=0.10; the fourth dimension: 0*0.7+0.33*0.3≈0+0.10=0.10; the fifth dimension: 121.78*0.7+121.30*0.3≈85.25+36.39=121.64; the sixth dimension: 4.44*0.7+4.10*0.3≈3.11+1.23=4.34; the seventh dimension: 126*0.7+125.33*0.3≈88.2+37.60=125.80; the eighth dimension: 120*0.7+119.33*0.3≈84+35.80=119.80; the ninth dimension: 123*0.7+122.33*0.3≈86.1+36.70=122.80.

[0147] The updated feature vector of the target node 2 is [0.10, 0.7, 0.10, 0.10, 121.64, 4.34, 125.80, 119.80, 122.80]. The above example is only an example of the present application, and other combination methods such as direct splicing and element-wise addition can also be used in actual application, which is not limited in the present application.

[0148] S1046, and the final updated node feature vector that satisfies the iteration termination condition is taken as the updated aggregated feature of the node.

[0149] The aggregated feature is the final feature vector formed after the complete iteration propagation and aggregation process of the node, which combines the initial feature of the node itself and the feature information transmitted by the adjacent nodes in the iteration process, and can more comprehensively and accurately represent the state of the spatial region corresponding to the node and the surrounding associated influence.

[0150] In the embodiments of the present application, the operations of S1043 to S1045 are continuously performed according to the preset iteration termination condition. If the termination condition is a fixed number of iterations, the last updated feature vector of each node is recorded after completing the preset number of iterations as the aggregated feature of the node; if the termination condition is that the feature change amount is less than a threshold value, the update change amount of the feature vector of all nodes is calculated after each iteration, and when the change amount meets the threshold requirement, the iteration is stopped and the feature vector of each node at this time is saved as the aggregated feature.

[0151] For example, continuing the previous example, the preset iteration termination condition is fixed iteration 3 times. The target node 2 updates the feature vector to [0.10, 0.7, 0.10, 0.10, 121.64, 4.34, 125.80, 119.80, 122.80] after the first iteration; in the second iteration, the neighborhood aggregation feature is recalculated based on the updated adjacent node features and fused, and the updated feature vector is [0.15, 0.5, 0.20, 0.15, 121.52, 4.28, 125.50, 119.60, 122.60]; after the third iteration, the final updated feature vector is [0.18, 0.4, 0.22, 0.20, 121.45, 4.25, 125.30, 119.50, 122.50]. The vector is the updated aggregation feature of the target node 2, which fuses the feature information of the node itself and its adjacent nodes in the three iterations.

[0152] The application realizes the deep fusion of the node's own features and the neighborhood associated features through the graph network modeling and the iterative propagation aggregation technology, overcomes the limitations of traditional single node feature analysis, can more comprehensively capture the associated influence of the surface space area of the cultural relics, and improves the accuracy and integrity of the feature representation.

[0153] Optionally, before the step of "mapping the material feature vector and the topographic feature vector corresponding to each space area to the initial features of the corresponding node in the pre-constructed graph network" in step S104, it further includes:

[0154] The graph network is constructed, and the graph network includes a plurality of nodes and edges connecting the nodes, wherein each node corresponds to a space area on the surface of the cultural relics, and the edges are constructed based on the adjacency relationship and feature similarity between the space areas.

[0155] The adjacency relationship refers to the geographical location association of different space areas on the surface of the cultural relics, and adjacent space areas have an adjacency relationship; the feature similarity refers to the closeness of the material feature vector and the topographic feature vector of different space areas, and the higher the closeness, the stronger the feature similarity; and the edge is the link connecting the nodes in the graph network, and is used to represent the association between the space areas corresponding to the nodes.

[0156] Specifically, first, according to the high-precision three-dimensional model of the cultural relics, the surface of the cultural relics is divided into a plurality of independent space areas according to a preset resolution, and each space area corresponds to a node in the graph network. Then, the adjacency relationship between the space areas is determined. If two space areas are adjacent in geographical position, they are preliminarily determined to be connectable. Then, the similarity of the feature vectors of the two space areas is calculated. If the similarity is higher than a preset threshold, an edge is constructed between the corresponding nodes; if only the adjacency relationship is met but the feature similarity does not meet the standard, an edge can also be constructed according to requirements to fully present the spatial association.

[0157] In another specific embodiment, the feature similarities between all spatial regions are calculated first, and region pairs with a similarity higher than a threshold are screened out, and then edges are constructed for the nodes of region pairs that are both adjacent and similar in feature, highlighting the core associations.

[0158] S105, generating a composite label for the spatial region corresponding to each node according to the updated aggregated feature of the node, the composite label containing material attributes, disease types and spatial position information of the spatial region.

[0159] Optionally, step S105 can specifically include the following steps:

[0160] The composite label is a comprehensive label integrating multi-dimensional key information of the spatial region, and can intuitively and centrally present the core state of the spatial region; the material attributes are the material composition and related characteristics of the spatial region, such as the degree of salt enrichment and water content; the disease type is the damage form existing in the spatial region, including categories such as sheet erosion, erosion, fissure, gully and biological damage; and the spatial position information is the specific geographic coordinates or relative position description of the spatial region on the cultural relic surface.

[0161] S1051, inputting the updated aggregated feature of each node in the graph network into a preset classification model to obtain the material attribute category and the disease type category corresponding to the node.

[0162] The preset classification model is an algorithm model pre-trained for identifying the category information corresponding to the feature vector, and can extract key information from the aggregated feature and match it to the corresponding material attribute category and disease type category.

[0163] In the embodiments of the present application, the classification model is constructed using a support vector machine algorithm. The model training process is as follows: first, a large number of aggregated feature samples labeled with material attribute categories and disease type categories are collected, and the samples are divided into a training set and a test set; then the support vector machine model is trained using the training set data, and the classification boundary is optimized by adjusting the model parameters, so that the model can accurately learn the corresponding relationship between the features and the categories; finally, the model performance is verified using the test set data, and when the classification accuracy reaches the preset standard, the model training is completed and put into use.

[0164] The updated aggregated feature of the node is input into the classification model, and the model determines the most matched material attribute category and disease type category of the feature vector by calculating the distance between the feature vector and the classification boundary of each category, thereby completing the analysis.

[0165] In another specific embodiment, a random forest algorithm can also be used to construct a classification model to improve the reliability of the classification results by using the integrated decision of multiple decision trees.

[0166] For example, continuing the previous example, the updated aggregated features of a certain node are [0.18, 0.4, 0.22, 0.20, 121.45, 4.25, 125.30, 119.50, 122.50]. The feature vector is input into the trained support vector machine classification model, and the model outputs the material attribute category as moderate salt enrichment rammed earth and the disease type category as a potential high-risk development area of erosion. The above example is only one example of the present application, and other algorithms can be selected to build a model according to the classification requirements in actual application, which is not limited in the present application.

[0167] S1052, obtaining spatial position information of the node.

[0168] The spatial position information is specific positioning data of the spatial region corresponding to the node in the cultural relic three-dimensional model, which can clearly indicate the orientation and coordinate range of the region on the surface of the cultural relic.

[0169] In a specific embodiment, relying on the high-precision three-dimensional model of cultural relics constructed in the early stage, each node is pre-associated with unique three-dimensional coordinate information (X, Y, Z) in the model. By querying the mapping relationship table of the node and the three-dimensional model coordinates, the three-dimensional coordinates corresponding to the node are directly extracted as the spatial position information; at the same time, combined with the overall structure division of the cultural relic, the three-dimensional coordinates are converted into a more understandable relative position description, such as the northeast side of the tower body, the middle, the west side of the bottom, etc.

[0170] For example, continuing the previous example, the mapping relationship table corresponding to the target node is queried, and the three-dimensional coordinates (35.2, 42.6, 12.8) are extracted, combined with the overall structure of the cultural relic, the coordinates are converted into a relative position description of the northeast side of the tower body, and the spatial position information of the node is finally determined as the three-dimensional coordinates (35.2, 42.6, 12.8) and the relative position of the northeast side of the tower body. The above example is only one example of the present application, and different position information representation methods can be selected according to the positioning accuracy requirements in actual application, which is not limited in the present application.

[0171] S1053, combining the material attribute category, the disease type category, and the extracted spatial position information to generate a composite label corresponding to the node.

[0172] The combination is a process of integrating the material attribute category, the disease type category, and the spatial position information in different dimensions according to a preset format to form a composite label with unified structure and complete information.

[0173] In a specific embodiment, the preset combined label has a combined format of "relative position-material attribute category-disease type category". According to the format, the extracted relative position information, the parsed material attribute category and the disease type category are sequentially spliced to generate the combined label. The format is concise and clear, and can quickly present the core information of the spatial region, facilitating the subsequent review and use of the protection work.

[0174] The application generates a combined label by parsing the classification model and combining multi-dimensional information, which can comprehensively and accurately integrate the key information of the spatial region, overcome the problem of single and fragmented information of the traditional label, realize the centralized and intuitive presentation of the information, and improve the convenience and effectiveness of the information use.

[0175] S106, constructing a knowledge graph of the target rammed earth cultural relic according to all generated combined labels.

[0176] The knowledge graph is a structured information display model, which takes entities as the core and correlation relationships as the link, and intuitively presents the logical correlation between different information. The preset knowledge graph framework is a pre-built empty structure model, which includes entity category definition, correlation relationship type division and other basic settings. The entity is a basic unit carrying information in the knowledge graph, which corresponds to material attributes, disease types, spatial position information and other specific contents. The initial attribute correlation is a basic correlation relationship between entities based on the logic of the combined label, which is used to clarify the correspondence between different types of entities. Optionally, step S106 can specifically include the following steps:

[0177] S1061, creating or matching corresponding entities for the material attributes, disease types and spatial position information parsed from the combined label in the preset knowledge graph framework.

[0178] The creation of the entity means that when the parsed information has no corresponding record in the knowledge graph framework, a new entity unit is added. The matching of the entity means that when the parsed information has a corresponding record in the knowledge graph framework, the existing entity is directly associated to avoid repeated creation.

[0179] In a specific embodiment, all combined labels are first batch parsed to extract the material attributes, disease types and spatial position information in each label. Then, for each type of information, the existing entities under the corresponding category in the preset knowledge graph framework are compared. The comparison adopts a string similarity matching algorithm to calculate the similarity between the parsed information and the existing entity name. When the similarity is higher than a preset threshold, it is determined that the matching is successful, and the existing entity is directly associated. When the similarity is lower than the preset threshold, it is determined that there is no corresponding entity, a new entity is created under the corresponding category, and the basic description information of the entity is improved.

[0180] For example, continuing the previous example, the information parsed from the composite label includes the material attribute of moderate salt-enriched rammed earth, the disease type of potential high-risk development area of erosion, and the spatial location of the middle part of the northeast side of the tower body. Comparing the material attribute of moderate salt-enriched rammed earth with existing entities under the material attribute category in the framework, if no entity with a similarity higher than 0.9 is found, a new entity of moderate salt-enriched rammed earth is created; if the disease type of potential high-risk development area of erosion already exists under the disease type category, the existing entity is directly matched; and since there is no corresponding entity for the spatial location of the middle part of the northeast side of the tower body, a new entity of the middle part of the northeast side of the tower body is created. The above example is only one example of the present application, and in actual application, the comparison algorithm and threshold can also be adjusted according to the type of information, which is not limited in the present application.

[0181] S1062, establishing initial attribute associations between entities representing spatial location information and entities representing material attributes and disease types.

[0182] The initial attribute association is based on the internal logic of the composite label and clearly defines the ownership relationship between the spatial location entity and the corresponding material attribute entity and disease type entity, which is the basic form of entity association in the knowledge graph.

[0183] In the embodiments of the present application, an association rule is established for each spatial location entity according to the corresponding relationship of the composite label. If a composite label contains a spatial location entity A, a material attribute entity B, and a disease type entity C, initial attribute associations of “the spatial location A has the material attribute B” and “the spatial location A exists the disease type C” are established between the three. After the association is established, it is presented in the form of a directed edge in the knowledge graph, and the label of the edge marks the association type, such as “has” and “exists”, so that the association relationship is clear and identifiable.

[0184] For example, continuing the example in the above step, the spatial location entity is the middle part of the northeast side of the tower body, the corresponding material attribute entity is moderate salt-enriched rammed earth, and the disease type entity is the potential high-risk development area of erosion. Based on the logic of the composite label, the initial attribute associations of “the middle part of the northeast side of the tower body has moderate salt-enriched rammed earth” and “the middle part of the northeast side of the tower body exists the potential high-risk development area of erosion” are established, and the corresponding entities are connected by directed edges with “has” and “exists” labels in the knowledge graph. The above example is only one example of the present application, and in actual application, more association types can be defined according to the complexity of information association, which is not limited in the present application.

[0185] S1063, integrating all created or matched entities and established initial attribute associations to complete the construction of the knowledge graph of the target rammed earth cultural relic.

[0186] Integration is the process of organizing and sorting out scattered entities and initial attributes according to a preset knowledge graph framework to form a logically coherent and structurally complete knowledge graph. This includes operations such as entity classification, association verification, and removal of redundant information.

[0187] In one specific implementation, all created or matched entities are first categorized into three types: material properties, disease type, and spatial location, and then assigned to the corresponding entity sets within the knowledge graph framework. Next, all initial attribute associations are validated to check their rationality and completeness, eliminating logically contradictory or duplicate relationships. Finally, entities and their relationships are presented graphically using visualization technology, with entities represented by nodes and relationships by directed edges, forming a complete knowledge graph that allows users to browse, query, and edit the graph.

[0188] For example, continuing the previous example, after the previous steps, we obtained multiple entities, including material properties such as moderately saline rammed soil and healthy dry rammed soil; disease types such as potential high-risk erosion development zones and sheet erosion; and spatial locations such as the northeast center of the tower and the northwest bottom of the tower, as well as their corresponding initial attribute associations. These entities are categorized into their corresponding sets, and after verifying that the associations are consistent, they are presented as a knowledge graph using visualization tools. Nodes are labeled with entity names, and edges are labeled with association types such as "has" or "exists." Users can intuitively see the association between the northeast center of the tower and the moderately saline rammed soil and the potential high-risk erosion development zone, as well as the associations between other spatial locations and their corresponding material properties and disease types.

[0189] For example, such as Figure 4 As shown, assuming that composite label 1 and composite label 2 are generated through the above steps, composite label 1 contains the material attribute "rammed earth A", the disease type "alkali efflorescence", and the spatial location "northeast upper-middle part", while composite label 2 contains the material attribute "rammed earth A", the disease type "alkali efflorescence", and the spatial location "northeast lower part". The entity material attributes "rammed earth A", disease type "alkali efflorescence", spatial location 1 (northeast upper-middle part), and spatial location 2 (northeast lower part) are parsed to obtain the entity material attributes "rammed earth A", "disease type "alkali efflorescence", and spatial location 1 (northeast upper-middle part) and 2 (northeast lower part) respectively. The relationships between the entities are marked according to the association between them: "has" and "exists".

[0190] The knowledge graph constructed in this application not only intuitively presents the core state of each area of ​​the cultural relic, but also clearly sorts out the logical connections between different areas. Compared with traditional manual recording and single-dimensional data statistics, it can more comprehensively restore the distribution pattern of cultural relic diseases and material properties, and provide more systematic and reliable decision support for risk prediction and restoration plan formulation in the preventive protection of cultural relics.

[0191] Optionally, after integrating all created or matched entities and established initial attribute associations to complete the construction of the knowledge graph of the target rammed earth cultural relic, the method further includes:

[0192] In the knowledge graph, a group of spatial location entities is identified, wherein all spatial location entities in the group are adjacent to each other in the graph network and jointly point to the same material attribute entity or the same disease type entity through the initial attribute association. A new aggregated entity is created for the group of spatial location entities, and the aggregated entity is integrated into the knowledge graph of the target rammed earth cultural relic. The aggregated entity is used to represent a continuous physical region formed by the group of spatial location entities.

[0193] In the above steps, the aggregated entity is a new entity created based on the common features and adjacency relationship of the group of spatial location entities, and is used to represent the continuous physical region formed by the group of spatial location entities, which can intuitively reflect the concentrated distribution of the same attribute or the same disease on the surface of the cultural relic.

[0194] In a specific implementation, first, all spatial location entities in the knowledge graph are traversed, and a group of spatial location entities that are adjacent to each other are selected in combination with the adjacency relationship in the graph network. Then, the initial attribute associations of all entities in the group are checked to determine whether they jointly point to the same material attribute entity or the same disease type entity. If the two conditions are met, an aggregated entity is created for the group of spatial location entities, and the name of the aggregated entity needs to reflect the attribute features of the continuous region, such as Figure 4 As shown, the disease types of the composite label 1 and the composite label 2 are both sponginess, a new aggregated entity is aggregated to reflect the existence of a continuous sponginess region, and finally the aggregated entity is added to the knowledge graph, and the association relationship between the aggregated entity and the corresponding spatial location entity, material attribute entity or disease type entity is established to complete the integration.

[0195] By generating the aggregated entity and supplementing it to the knowledge graph, it is beneficial to more comprehensively restore the disease distribution rule and material attribute features of the cultural relic, and to provide more systematic and reliable decision support for risk prediction, repair scheme development in preventive protection of cultural relics.

[0196] The tag-based multi-modal cultural relic knowledge graph construction method provided by the application realizes accurate association and structured presentation of rammed earth cultural relic material attributes, disease types and spatial position information through the whole process technical means of multi-modal data acquisition, feature fusion, graph network modeling and knowledge graph construction. It breaks through the limitations of traditional manual investigation, solves the problems of single data dimension deficiency and information fragmentation, and through intelligent algorithms and network models, it also excavates the correlation influence of the surface area of cultural relics, making disease identification and risk judgment more comprehensive and objective. At the same time, the structured knowledge graph makes the cultural relic state information intuitive and easy to understand, providing a clear decision basis for cultural relic protection work and promoting the transformation of ancient site preventive protection from experience-driven to data-driven.

[0197] Figure 5 A specific implementation structure diagram of a tag-based multi-modal cultural relic knowledge graph construction system provided by an embodiment of the application is shown in Figure 5 The system can include:

[0198] The acquisition module 51 is configured to acquire multi-modal data of the target rammed earth cultural relic, and the multi-modal data includes multi-spectral image data covering the surface of the cultural relic and texture image data corresponding to the texture of the surface of the cultural relic.

[0199] The determination module 52 is configured to perform spectral analysis on the multi-spectral image data based on a pre-constructed standard reflectance spectrum database of rammed earth materials to obtain material composition features corresponding to each spatial region on the surface of the cultural relic.

[0200] The generation module 53 is configured to associate the material composition features corresponding to each spatial region with the corresponding texture image data to generate a material feature vector and a topographic feature vector corresponding to each spatial region.

[0201] The processing module 54 is configured to map the material feature vector and the topographic feature vector corresponding to each spatial region to initial features of corresponding nodes in a pre-constructed graph network, and based on the initial features of the nodes and the connection relationship of the edges, perform iterative propagation and aggregation of node features on the graph network to obtain updated aggregated features of each node.

[0202] The generation module is further configured to generate a composite label for the spatial region corresponding to each node according to the updated aggregated features of each node, and the composite label contains material attributes, disease types and spatial position information of the spatial region.

[0203] The construction module 55 is configured to construct a knowledge graph of the target rammed earth cultural relic according to all generated composite labels.

[0204] The tag-based multi-modal cultural relic knowledge graph construction system of the embodiments of the present application is used to implement the tag-based multi-modal cultural relic knowledge graph construction method described above, and therefore the specific embodiments of the tag-based multi-modal cultural relic knowledge graph construction system can be seen from the embodiment part of the tag-based multi-modal cultural relic knowledge graph construction method described above. The specific embodiments can be referred to the description of the respective embodiment parts, and will not be described here.

[0205] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the tag-based multi-modal cultural relic knowledge graph construction methods described above when executing the computer program.

[0206] The present application also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any one of the tag-based multi-modal cultural relic knowledge graph construction methods described above.

[0207] In an exemplary embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0208] The embodiments of the present application also provide a computer program product, and the computer program product includes a computer program. The computer program is executed by a processor to implement the steps in any one of the tag-based multi-modal cultural relic knowledge graph construction method embodiments described above.

[0209] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0210] The tag-based multi-modal cultural relic knowledge graph construction method, system and device provided by the present application are described in detail above. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above example descriptions are only used to help understand the method and its core idea of the present application. It should be noted that the ordinary skilled person in the technical field can make some improvements and modifications to the present application without departing from the principles of the present application. These improvements and modifications also fall within the protection scope of the present application.

Claims

1. A label-based multi-modal cultural relic knowledge graph construction method, characterized in that, The method comprises the following steps: acquiring multi-modal data of a target rammed earth cultural relic, the multi-modal data comprising multi-spectral image data covering the surface of the cultural relic and texture image data corresponding to the texture of the surface of the cultural relic; performing spectral analysis on the multi-spectral image data based on a pre-constructed standard reflectance spectrum database of rammed earth materials to obtain material composition characteristics corresponding to each spatial region on the surface of the cultural relic; associating the material composition characteristics corresponding to each spatial region with the corresponding texture image data to generate a material feature vector and a topographic feature vector corresponding to each spatial region; mapping the material feature vector and the topographic feature vector corresponding to each spatial region to initial features of corresponding nodes in a pre-constructed graph network, and performing iterative propagation and aggregation of node features on the graph network based on the initial features of the nodes and the connection relationship of edges to obtain updated aggregated features of each node; generating a composite label for the spatial region corresponding to each node according to the updated aggregated features of the node, the composite label containing material attributes, disease types and spatial position information of the spatial region; constructing a knowledge graph of the target rammed earth cultural relic according to all generated composite labels; wherein the graph network comprises a plurality of nodes and edges connecting the nodes, wherein each node corresponds to a spatial region on the surface of the cultural relic, and the edges are constructed based on the adjacency relationship and feature similarity between the spatial regions; the step of performing iterative propagation and aggregation of node features on the graph network to obtain updated aggregated features of each node comprises: based on the initial features of each node, repeatedly performing the following propagation and aggregation operations until a preset iteration termination condition is met: for any target node in the graph network, acquiring current feature vectors of all adjacent nodes of the target node; aggregating the acquired current feature vectors of all adjacent nodes to generate neighborhood aggregated features; combining the current feature vector of the target node with the neighborhood aggregated features to update the feature vector of the target node; and taking the finally updated node feature vector as the updated aggregated features of the node when the iteration termination condition is met.

2. The method of claim 1, wherein, the step of generating a composite label for the spatial region corresponding to each node according to the updated aggregated features of the node comprises: for each node in the graph network, inputting the updated aggregated features of the node into a preset classification model to analyze and obtain the material attribute category and the disease type category corresponding to the node; acquiring spatial position information represented by the node; combining the analyzed material attribute category, the disease type category and the extracted spatial position information to generate the composite label corresponding to the node.

3. The method of claim 1, wherein, the step of constructing a knowledge graph of the target rammed earth cultural relic according to all generated composite labels comprises: creating or matching corresponding entities in a preset knowledge graph framework for the material attributes, disease types and spatial position information analyzed from the composite labels; establishing an initial attribute association between an entity representing the spatial position information and an entity representing the material attribute or the disease type; integrating all the created or matched entities and the established initial attribute association to complete the construction of the knowledge graph of the target rammed earth cultural relic.

4. The method of claim 3, wherein, After the step of integrating all the created or matched entities and the established initial attribute association to complete the construction of the knowledge graph of the target rammed earth cultural relic, the method further includes: identifying a group of spatial position entities in the knowledge graph, wherein all the spatial position entities in the group are adjacent to each other in the graph network and jointly point to the same material attribute entity or the same disease type entity through the initial attribute association; creating a new aggregate entity for the spatial position entities in the group and integrating the aggregate entity into the knowledge graph of the target rammed earth cultural relic, wherein the aggregate entity is used to represent a continuous physical region formed by the spatial position entities in the group.

5. The method of claim 1, wherein, The step of performing spectral analysis on the multispectral image data based on the pre-constructed standard reflectance spectrum database of rammed earth materials to obtain the material composition feature corresponding to each spatial region on the surface of the cultural relic includes: extracting a spectral response sequence of any spatial region in the multispectral image data; searching and determining a standard reflectance spectrum with the highest similarity to the spectral response sequence of the spatial region in the pre-constructed standard reflectance spectrum database of rammed earth materials; if the highest similarity is higher than a preset similarity threshold, determining the rammed earth material corresponding to the standard reflectance spectrum as the material composition feature of the spatial region.

6. The method of claim 1, wherein, The step of associating the material composition feature corresponding to each spatial region with the corresponding texture image data to generate a material feature vector and a topographic feature vector corresponding to each spatial region includes: encoding the material composition feature corresponding to each spatial region in the multispectral image data to generate a material feature vector of the spatial region; determining a neighborhood centered on the spatial region in the texture image data; statistically analyzing the texture value distribution in the neighborhood to generate a topographic feature vector of the spatial region.

7. The method of claim 1, wherein, The step of mapping the material feature vector and the topographic feature vector corresponding to each spatial region to the initial features of the corresponding nodes in the pre-constructed graph network includes: aggregating the material feature vector and the topographic feature vector of the spatial region corresponding to each node in the graph network to generate the initial features of the node. 8.A label-based multi-modal cultural relic knowledge graph construction system, configured to perform the label-based multi-modal cultural relic knowledge graph construction method according to any one of claims 1 to 7. The method includes: an acquisition module configured to acquire multi-modal data of a target rammed earth cultural relic, the multi-modal data including multispectral image data covering the surface of the cultural relic and texture image data corresponding to the texture of the surface of the cultural relic; a determination module configured to perform spectral analysis on the multispectral image data based on a pre-constructed standard reflectance spectrum database of rammed earth materials to obtain a material composition feature corresponding to each spatial region on the surface of the cultural relic; The generating module is configured to associate the material component feature corresponding to each spatial region with the corresponding texture image data, and generate a material feature vector and a topography feature vector corresponding to each spatial region; The processing module is configured to map the material feature vector and the topography feature vector corresponding to each spatial region into initial features of corresponding nodes in a pre-constructed graph network, and perform iterative propagation and aggregation of node features on the graph network based on the initial features of the nodes and the connection relationship of edges, to obtain updated aggregated features of each node; The generating module is further configured to generate a composite label for the spatial region corresponding to each node according to the updated aggregated features of each node, and the composite label contains material attributes, disease types and spatial position information of the spatial region; The constructing module is configured to construct a knowledge graph of the target rammed earth cultural relic according to all generated composite labels.

9. An electronic device, comprising: The computer program is stored in the memory and includes a plurality of instructions executable by the processor. The processor is configured to execute the computer program to implement the steps of the label-based multi-modal cultural relic knowledge graph construction method according to any one of claims 1 to 7. ​

Images