Knowledge completion method and device for natural collections

Abstract

The embodiment of the invention aims to provide a knowledge completion method and equipment for natural collections, a computer program product and a computer program storage medium. Wherein a training data set used for the knowledge completion model is constructed based on multi-modal data in the field of the natural museum, and each training sample is an attribute data set of a natural collection; the knowledge completion model is trained based on the training data set, and a training task comprises preset attribute completion and extended attribute generation; known attribute data of a target collection is input into the trained knowledge completion model to generate a candidate attribute data set of the target collection, and the candidate attribute data set comprises missing preset attribute data and extended attribute data; performing rule verification and expert auditing on the candidate attribute data set to obtain supplementary attribute data of the target collection; and inserting the supplementary attribute data into the knowledge graph of the target collection, and recording a complementation source.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of museum information systems and management as well as knowledge graph technology, and in particular to a knowledge completion technology for natural history collections. Background Technology

[0002] In the digitization process of natural history museums, constructing a knowledge graph for natural history collections faces the core challenge of fragmented and dispersed raw data. Structured data in existing collection management systems (such as specimen numbers and classification information) often suffers from missing fields and inconsistent terminology; for example, the scientific and common names of the same species are used interchangeably, and descriptions of collection locations are vague (e.g., "Western Sichuan" lacks latitude and longitude coordinates). The deep knowledge (such as ecological habits and geological time relationships) contained in semi-structured data (such as popular science texts and research papers) and unstructured data (such as specimen images and collection recording videos) is difficult to extract effectively from traditional databases.

[0003] Therefore, in order to solve these problems, this disclosure proposes a knowledge completion scheme for natural collections. Summary of the Invention

[0004] The purpose of the various embodiments disclosed herein is to provide a method, apparatus, computer program product, and computer program storage medium for knowledge completion of natural collections.

[0005] According to one aspect of this disclosure, a method for knowledge completion in the field of natural collections is provided, wherein the method includes the following steps: A training dataset for a knowledge completion model is constructed based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection. The knowledge completion model is trained based on the training dataset, wherein the training tasks include preset attribute completion and extended attribute generation; The known attribute data of the target collection is input into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data. The candidate attribute dataset is subjected to rule validation and expert review to obtain supplementary attribute data for the target collection. The supplementary attribute data is inserted into the knowledge graph of the target collection, and the source of the supplement is recorded.

[0006] According to one aspect of this disclosure, a knowledge completion device for natural collections is also provided, wherein the device includes a memory and a processor, the memory storing computer program instructions, and when the computer program instructions are executed by the processor, the device is configured to perform the following operations: A training dataset for a knowledge completion model is constructed based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection. The knowledge completion model is trained based on the training dataset, wherein the training tasks include preset attribute completion and extended attribute generation; The known attribute data of the target collection is input into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data. The candidate attribute dataset is subjected to rule validation and expert review to obtain supplementary attribute data for the target collection. The supplementary attribute data is inserted into the knowledge graph of the target collection, and the source of the supplement is recorded.

[0007] According to one aspect of this disclosure, a computer program product is also provided, including computer program instructions, wherein when the computer program instructions are executed by a computer device, the computer device is configured to perform a knowledge completion method for natural collections, wherein the method includes the following steps: A training dataset for a knowledge completion model is constructed based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection. The knowledge completion model is trained based on the training dataset, wherein the training tasks include preset attribute completion and extended attribute generation; The known attribute data of the target collection is input into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data. The candidate attribute dataset is subjected to rule validation and expert review to obtain supplementary attribute data for the target collection. The supplementary attribute data is inserted into the knowledge graph of the target collection, and the source of the supplement is recorded.

[0008] According to one aspect of this disclosure, a computer program storage medium is also provided, wherein computer-executable instructions are stored, and when the computer-executable instructions are executed by a computer device, the computer device is configured to perform a knowledge completion method for natural collections, wherein the method includes the following steps: A training dataset for a knowledge completion model is constructed based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection. The knowledge completion model is trained based on the training dataset, wherein the training tasks include preset attribute completion and extended attribute generation; The known attribute data of the target collection is input into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data. The candidate attribute dataset is subjected to rule validation and expert review to obtain supplementary attribute data for the target collection. The supplementary attribute data is inserted into the knowledge graph of the target collection, and the source of the supplement is recorded.

[0009] The embodiments disclosed herein significantly improve the efficiency and accuracy of knowledge graph construction by employing multimodal data fusion and intelligent processing when performing knowledge completion on natural artifacts. First, a training dataset is constructed based on multimodal data (such as structured attributes, images, videos, and external authoritative databases) to ensure the model can comprehensively learn the complex relationships between artifact attributes, such as inferring preservation status through specimen image features or verifying the rationality of the collection site by combining climate data. Second, a lightweight adaptation of the pre-trained large language model is performed using efficient parameter fine-tuning techniques (such as LoRA). While retaining general language understanding capabilities, only a small number of parameters need to be trained to accurately capture domain characteristics, such as associating the classification of "Carnivora" with the pattern of "mountainous distribution," significantly reducing computational resource consumption.

[0010] During the candidate attribute generation phase, the model not only completes the preset missing attributes (such as collection time and protection level) but also dynamically generates extended attributes (such as climate adaptability and oxidation risk), breaking through the limitations of traditional database fields. For example, given the input "Specimen name: Crested Ibis, Protection level: National Class I," the model generates "Endangered factors: Habitat reduction, Protection recommendation: Wetland ecological restoration." Through automated verification by the rule engine and external databases (such as GBIF and WorldClim), a large number of logical conflicts (such as "Polar species - Tropical collection site") are intercepted. The remaining disputed data is pushed to the expert review platform, and can be combined with visual evidence (such as species distribution maps and climate heat maps) to assist human decision-making, reducing the workload of manual review. The supplementary attributes that pass the review are updated to the knowledge graph in real time, and the data source and correction trajectory are recorded to ensure traceability and compliance.

[0011] Through full-process automation and dynamic learning mechanisms, this disclosure enables the continuous expansion and optimization of knowledge about natural collections. For example, the model automatically reveals implicit patterns such as "high-altitude specimens are more susceptible to temperature differences," providing new insights for conservation strategies for natural collections. Attached Figure Description

[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1A flowchart illustrating a knowledge completion method for natural collections according to one embodiment of the present disclosure is shown.

[0013] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0014] The specific embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0015] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments of this disclosure are described as apparatuses represented by block diagrams and processes or methods represented by flowcharts. Although the flowcharts depict the operation processes of the various embodiments of this disclosure as sequential processes, many of the operations may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The processes of the various embodiments of this disclosure may be terminated when their operations are completed, but may also include additional steps not shown in the flowcharts. The processes of the various embodiments of this disclosure may correspond to methods, functions, procedures, subroutines, subroutines, etc.

[0016] The methods illustrated in the flowcharts and the apparatuses illustrated in the block diagrams discussed below can be implemented in hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks can be stored in a machine or a computer-readable medium such as a storage medium. One or more processors can perform the necessary tasks.

[0017] Similarly, it will also understand any flowchart, state transition diagram, and the like, representing various processes that can be adequately described as program code stored in a computer-readable medium and thus executed by a computer device or processor, whether or not such computer device or processor is explicitly shown.

[0018] In this document, the term "storage medium" can refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing and / or containing instructions and / or data.

[0019] A code segment can represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program descriptions. A code segment can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or stored content. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any suitable means, including storage sharing, message passing, token passing, network transmission, etc.

[0020] In this context, "computer device" refers to an electronic device that can perform predetermined processing procedures such as numerical calculations and / or logical calculations by running predetermined programs or instructions. It may include at least a processor and a memory, wherein the predetermined processing procedures are performed by the processor executing program instructions pre-stored in the memory, or by hardware such as ASIC, FPGA, DSP, etc., or by a combination of the above.

[0021] The term "computer device" as used above is generally embodied in the form of a general-purpose computer device, whose components may include, but are not limited to, one or more processors or processing units and system memory. System memory may include computer-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The "computer device" may further include other removable / non-removable, volatile / non-volatile computer-readable storage media. The memory may include at least one computer program product having a set (e.g., at least one) of program modules configured to perform the functions and / or methods of the embodiments of this disclosure. The processor executes various functional applications and data processing by running programs stored in the memory.

[0022] For example, a computer program for performing various functions and processes of multiple embodiments of the present disclosure is stored in the memory, and when the processor executes the corresponding computer program, the digital resource acquisition system of the present disclosure is implemented.

[0023] Typically, computer devices can be user devices or network devices, or even a combination of both. User devices include, but are not limited to, personal computers (PCs), laptops, and mobile terminals; mobile terminals include, but are not limited to, smartphones and tablets. Network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing, which is a type of distributed computing consisting of a super virtual computer composed of a group of loosely coupled computers. The computer devices can operate independently to implement the embodiments of this disclosure, or they can connect to a network and implement the embodiments of this disclosure through interaction with other computer devices in the network. The networks in which the computer devices reside include, but are not limited to, the Internet, wide area networks (WANs), metropolitan area networks (MANs), local area networks (LANs), and VPN networks.

[0024] It should be noted that the user equipment, network equipment, and network mentioned are merely examples. Other existing or future computing devices or networks that are applicable to the embodiments of this disclosure should also be included within the scope of protection of this disclosure and are incorporated herein by reference.

[0025] The specific structural and functional details disclosed herein are merely representative and are intended to describe exemplary embodiments of this disclosure. However, the various embodiments of this disclosure can be implemented in many alternative forms and should not be construed as being limited solely to the embodiments set forth herein.

[0026] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments. Unless the context clearly indicates otherwise, the singular forms “a” and “an” as used herein are also intended to include the plural. It should also be understood that the terms “comprising” and / or “including” as used herein specify the presence of the stated features, integers, steps, operations, units, and / or components, without excluding the presence or addition of one or more other features, integers, steps, operations, units, components, and / or combinations thereof.

[0028] It should also be mentioned that in some alternative implementations, the functions / actions mentioned may occur in a different order than those shown in the figures. For example, depending on the functions / actions involved, the two figures shown successively may actually be executed substantially simultaneously or sometimes in reverse order.

[0029] See Figure 1 The document illustrates a knowledge completion process for natural collections according to one embodiment of the present disclosure.

[0030] like Figure 1 As shown, in step S1, the knowledge completion system constructs a training dataset for the knowledge completion model based on multimodal data from the natural history museum field; in step S2, the knowledge completion system trains the knowledge completion model based on the training dataset; in step S3, the knowledge completion system inputs the known attribute data of the target collection into the trained knowledge completion model to generate a candidate attribute dataset for the target collection, including missing preset attribute data and extended attribute data; in step S4, the knowledge completion system performs rule verification and expert review on the candidate attribute dataset to obtain supplementary attribute data for the target collection; in step S5, the knowledge completion system inserts the supplementary attribute data into the knowledge graph of the target collection and records the source of the completion.

[0031] This publicly available knowledge completion system is an independent functional module, but it can also be used as a subsystem of the knowledge graph of natural history collections in natural history museums to supplement the attribute data of the collection entities.

[0032] According to one example, the natural collections in this disclosure include five major categories: plants, animals, paleontology, geology, and humanities, covering multiple disciplines.

[0033] Specifically, in step S1, the knowledge completion system constructs a training dataset for the knowledge completion model based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection.

[0034] In some embodiments, multimodal data in the field of natural history museums includes not only data related to the museum's internal collections but also data from external databases. Specifically: Internal data 1) Structured data: such as data from collection management systems (e.g., specimen number, classification, collection time, collection location, conservation level).

[0035] 2) Semi-structured data: such as popular science content about the collection (collection introduction in HTML / XML format).

[0036] 3) Unstructured data: such as high-resolution images of collections (specimen photos), videos related to collections (records of the acquisition process), and research papers related to collections (PDF).

[0037] External authoritative databases

[0038] 1) GBIF (Global Biodiversity Information) can provide the following data: Species distribution record: Records of species discoveries worldwide (time, location, latitude and longitude); Taxonomic information: The complete taxonomic chain of a species (phylum, class, order, family, genus, species); Collect metadata: specimen collector, collection time, storage institution, etc.; Ecological relationships: the relationship between species and their habitats and symbiotic organisms.

[0039] For example, if a giant panda specimen is missing from its collection site, a GBIF query for "giant panda (…)" can be used to find the relevant information. Ailuropoda melanoleuca melanoleuca Based on the distribution records of “)”, it can be inferred that it may have been collected from Sichuan, Shaanxi and other places.

[0040] 2) WorldClim (climate data) can provide data on the following climate variables: Temperature (annual average temperature, temperature of the coldest / hottest month); Precipitation (annual precipitation, seasonal precipitation distribution); Nineteen bioclimatic variables (such as temperature seasonality and precipitation variability).

[0041] For example, a plant specimen was collected in Xishuangbanna, Yunnan. The average annual temperature (22°C) and annual precipitation (1500 mm) of that area were obtained through WorldClim and used as input features for the model completion to infer the potential distribution of other similar climate regions.

[0042] 3) GeoNames (geographic data) can provide the following data: Geocoding: Converting text place names (such as "Wolong, Sichuan") into latitude and longitude coordinates; Geographical attributes: altitude, administrative division, landform type (mountains, plains, etc.); Multilingual support: Multilingual aliases and standardized names for place names worldwide.

[0043] For example, the location of a geological specimen is described as "northeastern Tibetan Plateau". GeoNames is used to resolve the specific latitude and longitude (e.g., 36.0°N, 94.0°E) and obtain the altitude (4500 meters) and landform type (plateau).

[0044] Here, GeoNames plays a crucial role in the preprocessing stage by providing a foundation for terminology standardization. By unifying geographic coordinates and attributes, it provides accurate input for subsequent analyses (such as climate inference and species distribution verification). Furthermore, GeoNames can also be used to generate extended attributes, such as supplementary geographic attributes like altitude and landform, directly supporting ecological modeling (such as "high-altitude species adaptability analysis").

[0045] The synergistic effect of WorldClim and GBIF in the prediction phase includes dynamic reasoning and validation. For example, after WorldClim generates climate attributes, GBIF can further validate their match with species distribution. For instance, if a region's climate is suitable for tropical species, but GBIF has no distribution records, this indicates a contradiction. Furthermore, by integrating the validation results of WorldClim and GBIF through a rule engine, unreasonable predictions (such as "hardy species - tropical climate") are automatically filtered out, reducing the burden of manual review.

[0046] Therefore, GeoNames completes geographic standardization and basic attribute expansion during the corpus preprocessing stage, providing structured input for subsequent processes. WorldClim generates climate-related extended attributes during the attribute prediction stage and participates in ecological rationality verification. GBIF can participate in the entire process to verify species distribution and attribute logic, ensuring the scientific consistency of the data. The three work together to form a "geography-climate-ecology" closed loop, realizing full-link automation from data cleaning and attribute supplementation to dynamic verification, significantly improving the efficiency and accuracy of knowledge completion, while balancing efficiency and authority through human-machine collaboration.

[0047] In some embodiments, each training sample in the training dataset corresponds to a set of attribute data for a natural collection.

[0048] As an example, based on industry-specific database classification standards, a list of attributes is predefined for each category, such as name, category, collection time, collection location, and conservation level. Each artifact is then used to extract corresponding attribute data from a corpus of natural artifacts according to this attribute list. Each sample represents a set of attribute data for one artifact, which may include all attributes defined in the attribute list, or only some of them. The attribute list can be dynamically expanded as needed, such as adding attribute fields like "preservation status" and "related documents."

[0049] When constructing attribute data for natural history collections based on data from the museum's internal collections, it's necessary to integrate multi-source, heterogeneous data and form a unified semantic framework through structured processing and knowledge extraction. First, structured data (such as specimen numbers and classification information) is cleaned and standardized before being directly mapped to attribute fields. For example, the collection time format is standardized to "YYYY-MM-DD", collection names are mapped to the same terminology, and collection locations are mapped to latitude and longitude to ensure data consistency. Semi-structured data (such as HTML-formatted science content) extracts key information by parsing tags, for example, from "..." <species> Giant panda< / species> Species names are extracted from the data and associated with industry-specific subject database classification standards. For unstructured text data (such as research papers), ecological and behavioral paragraphs of species are extracted from PDF papers.

[0050] When extracting domain knowledge from text, a triple structure needs to be constructed to clarify the relationships between entities. For example, extracting "giant panda-habitat-Qinling Mountains" from a research paper transforms unstructured text into a "subject-relationship-object" format. The triple structure, by capturing semantic associations between attributes, supports the construction of knowledge graphs and complex queries. For instance, based on a triple of "collection-collection location-climate type," the species distribution patterns of a specific region can be inferred. This process requires combining a rule engine and a machine learning model to ensure the accuracy and coverage of the extraction, such as locating species names through named entity recognition and extracting relational predicates through dependency parsing.

[0051] The resulting attribute dataset integrates structured fields, semi-structured extraction results, and triple knowledge to form a multidimensional semantic network. For example, the attribute data of a giant panda specimen includes basic information (number, classification), image descriptions (shoulder color, limb color), climate data of the collection site (temperature, precipitation), and associated triples ("giant panda-diet-bamboo"). This integrated representation not only improves data utilization but also provides underlying support for model training and knowledge services, such as completing missing attributes through triple path reasoning.

[0052] Furthermore, the data provided by the aforementioned authoritative external databases can be encoded with climate and geographic features. Climate and geographic data can be used as structured fields and concatenated with textual descriptions (such as species habits) to obtain a set of attribute data for the collection. In other words, through multimodal data integration, the multimodal input to the knowledge completion model becomes a joint input of text, climate, and geography.

[0053] Climate feature coding may include: Continuous values: Directly use the numerical values ​​of climate variables (such as temperature and precipitation).

[0054] Classification coding: K-means clustering is used to group multidimensional climate variables and generate climate type labels, such as "temperate monsoon climate" and "tropical rainforest climate".

[0055] Geographic feature coding can include: Continuous values: latitude and longitude, altitude, slope, etc.

[0056] Classification code: landform type (e.g., mountain = 1, plain = 2).

[0057] For example, using the specimen collection location (e.g., "Wolong, Sichuan"), latitude, longitude, and altitude are obtained via GeoNames. Based on latitude and longitude, climate variables (such as temperature and precipitation) are extracted from WorldClim. The reasonableness of the species distribution record at that latitude and longitude is verified using GBIF.

[0058] Accordingly, by linking the latitude and longitude of the collection sites to external climate and geographical databases, continuous / categorical variables can be encoded into a model-understandable format, allowing climate and geographical data to be integrated into the sample dataset. For example, climate data (e.g., Bio1 = average annual temperature) and geographical data (e.g., altitude) can be added to the sample set as new attribute data. The introduction of climate and geographical data can enhance the model's understanding of the attributes of natural collections. For instance, climate data can be used to infer species adaptability (e.g., "temperate species"), and altitude and landform type can be used to predict collection difficulty (e.g., "species from high-altitude areas are rarer"). The final sample set will contain multimodal features such as text, numerical values, and categories. For example, a "species-climate-geography" association matrix can be established, providing rich contextual information for large language models and improving the accuracy and scientific rigor of attribute completion. This multi-source data fusion not only enriches the sample features but also provides the model with more comprehensive background information on natural collections, significantly improving the accuracy and practicality of the attribute completion task.

[0059] Each collection's training sample consists of preset attributes and optional extended attributes, allowing for missing fields to simulate the incompleteness of real-world data.

[0060] Preset attributes: Core fields (such as specimen number, classification, collection time, and location) may partially exist, and missing attributes are used as learning targets for the model.

[0061] Extended attributes: generated from external data (such as WorldClim climate types) or triple inference, as additional labels (optional).

[0062] In step S2, the knowledge completion system trains the knowledge completion model based on the constructed training dataset.

[0063] Each training sample does not need to contain the complete set of preset attributes. The core capability of the knowledge completion model lies in inferring missing information from known attributes. For example, if a sample only contains "Name = Giant Panda Subadult Skull, Collection Altitude = 2650 meters, Associated Plants = Arrow Bamboo, Collector = Pan Wenshi Expedition Team", but lacks fields such as "Collection Location", the knowledge completion model needs to predict the missing value based on the existing attributes. To achieve this goal, a masking training strategy is adopted, that is, during the training process, some preset attributes are randomly masked (such as replacing the "Collection Location" field with a placeholder), constructing a mapping relationship of "known attributes → missing attributes". For example, the input sample may be constructed as "Name: Giant Panda Subadult Skull, Collection Altitude: 2650 meters, Associated Plants: Arrow Bamboo, Collector: Pan Wenshi Expedition Team, Collection Location: ______", and the goal of the knowledge completion model is to generate "Collection Location: Wolong National Nature Reserve, Sichuan". In this way, the knowledge completion model learns to capture the implicit relationships between attributes from incomplete data, rather than relying on the full set of preset attributes. The masking strategy not only simulates the incompleteness of data in real-world scenarios, but also enhances the reasoning ability of the knowledge completion model for complex relationships. This allows it to flexibly complete missing fields and generate extended attributes (such as climate adaptability or protection recommendations) during prediction, thereby improving the dynamic scalability and practicality of the knowledge graph.

[0064] The training task of the knowledge completion model is to generate missing preset attribute data and generate extended attribute data.

[0065] Based on one example, this can be treated as a single-task learning (uniform training), where pre-defined attribute completion and extended attribute generation are considered the same task. The knowledge completion model directly outputs all candidate attributes (including pre-defined and extended attributes). For example, after inputting known attributes, the model simultaneously generates "collection location (pre-defined)" and "ecological adaptability (extended)".

[0066] For example, enter: Name: Giant Panda Subadult Skull Collection altitude: 2650 meters Associated plants: Arrow bamboo Collected by: Pan Wenshi Scientific Expedition Team Output: Collection location: Wolong, Sichuan Ecological adaptability: Cold-resistant In another example, the knowledge completion model treats preset attribute completion and extended attribute generation as two independent tasks, sharing the model's underlying parameters but processed through different output headers. For example: Task 1: Input known attributes → Output missing preset attributes.

[0067] Task 2: Input known attributes → Output extended attributes (such as ecological adaptability).

[0068] For example, enter: Name: Giant Panda Distribution: Sichuan, Shaanxi, Gansu Specimen type: Subadult skull Collection time: 1983 Associated plants: Arrow bamboo Altitude: 2650 meters Task 1 Output: "Collection Location: Wolong National Nature Reserve, Sichuan" Task 2 output: "Ecological adaptability: cold resistance".

[0069] The knowledge completion model predicts both preset attributes and extended attributes simultaneously, and the loss function calculates the error between the two in a weighted average.

[0070] During training, the knowledge completion model's learning objectives include not only completing preset attributes (such as collection location and protection level), but also implicitly learning the association rules between attributes, laying the foundation for the generation of extended attributes. The specific implementation is as follows: Input design: The known attributes include preset attributes and some extended attributes (such as ecological adaptability).

[0071] For example, the input sample is: "Name: Giant Panda subadult skull, collection altitude: 2650 meters, associated plants: arrow bamboo → missing attribute: ecological adaptability".

[0072] Output target: Explicit goals: Complete preset attributes (such as "ecological adaptability").

[0073] Implicit learning: Relationships between attributes (such as the association between "ecological adaptability: cold resistance" and "collection location: Wolong National Nature Reserve, Sichuan").

[0074] The knowledge completion model then outputs extended attributes during the prediction phase through the following mechanism: 1) Attribute-related reasoning: The model generates logically related extended attributes based on known attributes and rules learned during training. For example, given the input "Name: Giant Panda subadult skull, collection altitude: 2650 meters, associated plant: Arrow bamboo", the knowledge completion model outputs "Collection location: Wolong National Nature Reserve, Sichuan".

[0075] 2) Engineering guidance prompts: The model can also embed extended attribute generation instructions in the input template. For example: "Complete missing attributes and infer ecological adaptability: {known attributes} → Output: {collection location:; ecological adaptability:}".

[0076] 3) Implicit fusion of external knowledge: The training data integrates implicit associations from external databases (such as WorldClim climate data), and the model learns these patterns through fine-tuning. For example, learning "average annual temperature 15°C → temperate climate → adapted species: giant panda".

[0077] The data sources for extended attributes include domain knowledge injection, external database association, and unstructured text mining. Using databases such as GeoNames and WorldClim, geographic coordinates are bound to climate and landform data to automatically generate extended attribute labels. For example, latitude and longitude 103.0°E, 31.0°N are associated with "average annual temperature 15°C, landform = mountainous". Entities and relationships are extracted from research papers and popular science content using a NER model to generate triplet knowledge (such as "giant panda - diet - bamboo"), which is then added to the training data as extended attributes.

[0078] As an example, the knowledge completion model includes a large textual language model. Therefore, the multimodal data of the collection needs to be preprocessed into text and a text sample set needs to be constructed.

[0079] For example, for image data of collections, image features such as shape, color, and texture are extracted using the ResNet model. The CLIP model is then used to map the image and text to the same semantic space through contrastive learning to obtain the corresponding text description, such as "the specimen has black stripes on its back".

[0080] For video data of collections, keyframes are extracted, image features are extracted frame by frame and corresponding text descriptions are generated, or a video description model (such as VideoBERT) is used to generate an overall summary of the video.

[0081] The text descriptions generated by the model are merged with the original structured data (such as collection time and location) to form a pure text training set. Training samples containing known and missing attributes are constructed. For example, a sample retains "Name = Giant Panda Subadult Skull, Collection Altitude = 2650 meters, Associated Plants = Arrow Bamboo", with the "Collection Location" field masked as the prediction target. This yields a standardized training dataset, where each sample contains some known attributes and missing attributes to be completed. The structured attribute data is converted into text, for example, attribute key-value pairs are converted into natural language descriptions. Specifically, "Name: Giant Panda Subadult Skull, Collection Altitude: 2650 meters, Associated Plants: Arrow Bamboo" is converted into the text: "Specimen Name: Giant Panda Subadult Skull, Collection Altitude: 2650 meters, Associated Plants: Arrow Bamboo, Please complete the collection location." The model is guided to focus on the task through prompt engineering; for example, the template "Please complete the missing field based on the following attributes: {Known Attributes} → Missing Attribute: Collection Location" explicitly requires the model to generate the target field. After prompt word engineering, the output is a well-structured text sequence containing task instructions, such as "Name: Giant Panda Subadult Skull, Collection Altitude: 2650 meters, Associated Plants: Arrow Bamboo → Collection Location: ______", which can be directly processed by the model.

[0082] Knowledge completion models employ pure text-based large language models, such as LLaMA-2 and GPT. These models (like LLaMA and GPT) are based on the Transformer architecture, and their core capability is to understand and generate natural language by learning linguistic patterns from large-scale text data. The core components of the model include an input embedding layer, multi-layer Transformer modules, and an output layer. The input embedding layer converts the text into a high-dimensional vector representation, enabling the computer to process semantic information. The multi-layer Transformer modules capture long-distance dependencies between words through a self-attention mechanism, such as identifying the taxonomic association between "giant panda" and "arrow bamboo." The feedforward network further performs a non-linear transformation on the attention output, enhancing the model's expressive power. Residual connections and layer normalization techniques ensure the stability of the training process, avoiding gradient vanishing or exploding problems. The output layer generates the probability distribution of words, used to predict the next word or missing fields.

[0083] The training process is divided into two stages: pre-training and fine-tuning.

[0084] The pre-training phase utilizes massive amounts of unsupervised text data (such as books and web pages) to learn language patterns through autoregressive language modeling tasks (predicting the next word) or masked language modeling tasks (predicting masked words). For example, the input text "Specimen collection location: Sichuan" is segmented into a sequence of lexical units. The model predicts the masked "Sichuan" based on the context, thus understanding the association between "collection location" and geographical names. This phase enables the model to possess general language understanding capabilities, such as understanding grammatical structures, terminology definitions, and domain knowledge.

[0085] Load all parameters of the pre-trained model and fix them to preserve its general language understanding capabilities. For example, the original parameters of LLaMA-2 are not updated during training to avoid disrupting the basic language rules learned from massive amounts of data due to fine-tuning.

[0086] The fine-tuning phase adapts the system for attribute completion tasks related to natural artifacts. Input data consists of pre-processed training samples containing known attributes (e.g., "Name: Giant Panda Subadult Skull, Collection Altitude: 2650 meters, Associated Plants: Arrow Bamboo") and masked missing attributes (e.g., "Collection Location: ______"). Through prompting engineering, the structured attributes are converted into natural language instructions, such as "Please complete the following attributes: Name: Giant Panda Subadult Skull, Collection Altitude: 2650 meters, Associated Plants: Arrow Bamboo → ______" The "Collection Location:" directs the model to focus on the task objective. During training, efficient parameter fine-tuning techniques (such as LoRA) are employed, freezing most of the parameters of the pre-trained model. Low-rank adaptation matrices are inserted only at the self-attention layer. For example, the original weight matrix is ​​decomposed into two small-dimensional matrices, significantly reducing the number of parameters (e.g., to 1% of the original size). For instance, a matrix with the original dimension of 4096×4096 is decomposed into two matrices of 4096×64 and 64×4096. The optimizer is set to update the gradients of the LoRA parameters only, while the base parameters remain frozen. Through multiple iterations, the model learns the mapping pattern from known attributes to missing attributes, and finally outputs the completed attribute data (e.g., "Collection Location: Wolong National Nature Reserve, Sichuan").

[0087] The lightweight model architecture for attribute completion tasks retains the general capabilities of the base model while adapting to the characteristics of the natural collection domain with a small number of trainable parameters. For example, the model can understand the association between "ecological adaptability: cold resistance" and "collection location: Wolong National Nature Reserve, Sichuan" without having to relearn the basic grammar rules.

[0088] Using LoRA technology, models can achieve high-precision completion with only 1% of the model parameters, saving significant computational resources and enabling small and medium-sized institutions to deploy large-scale models at low cost. While retaining general language capabilities, the model accurately learns the domain patterns of natural collections. For example, inputting "Mamenchisaurus fossil, Late Jurassic," the model infers "Collection strata: Late Jurassic terrestrial red beds," supporting paleontological research. Dynamically extended attributes can break through preset attribute limitations, generating extended attributes such as climate adaptability and conservation recommendations. For example, inputting "Specimen name: Crested Ibis, Conservation level: National Class I," the model generates "Endangered factors: Habitat reduction, Conservation recommendation: Wetland ecological restoration." During training, the model automatically captures non-explicit associations. For example, it reveals patterns such as "High-altitude specimens are more susceptible to temperature differences and require constant temperature preservation," providing new perspectives for collection conservation.

[0089] As an example, a knowledge completion model can employ a cascaded structure. For instance, an image / video description generation model (such as the CLIP model) first generates text descriptions for image data. The resulting text sample data is then input into a large text language model (such as LLaMA-2) for training. In this example, the CLIP model is no longer a preprocessor of the image data but rather serves as part of the knowledge completion model, generating text descriptions for the image data input to the model. Therefore, the knowledge completion model in this example can directly accept multimodal data input. To improve training efficiency, LoRA techniques can be applied to the image encoder of CLIP and the self-attention layer of LLaMA-2, respectively. Insert low-rank matrices into the self-attention layer of the CLIP image encoder, for example, decompose the original weight matrix into two small-dimensional matrices, and train only these parameters to optimize the image-to-text generation capability, making it more suitable for the visual characteristics of natural collections (such as fossil textures and specimen preservation status).

[0090] A low-rank matrix is ​​inserted into the text encoding layer of LLaMA-2, the pre-trained parameters are frozen, and only a small number of parameters are adjusted to learn the mapping rules from multimodal concatenated text (image description + structured attributes) to missing attributes.

[0091] This cascaded structure achieves lightweight fine-tuning of the two modules through LoRA. The refined descriptions generated by CLIP (such as "the specimen's back fur is black") are optimized in synergy with the attribute reasoning ability of LLaMA-2. The model can directly accept mixed input of images and text and output the completed attribute set.

[0092] As an example, the knowledge completion model includes a multimodal large language model. Thus, after aligning image and video data from the collection's multimodal data with text, the aligned feature information can be extracted and used to construct a multimodal sample set along with other structured data.

[0093] Among them, image-text alignment involves labeling each image with relevant attribute text, such as "Collection location: Wolong, Sichuan".

[0094] Video-text alignment involves extracting keyframes from the video and associating timestamps with descriptive text, such as "10 seconds: showing the back of the specimen." Alternatively, a 3D CNN or ViViT can be used to encode the video frame sequence, outputting spatiotemporal embedding vectors into a multimodal large language model.

[0095] The knowledge completion model uses a multimodal large language model, such as Flamingo or BLIP-2, which supports simultaneous input of text and image / video features.

[0096] Multimodal large language models (such as Flamingo and BLIP-2) achieve cross-modal semantic understanding and generation by fusing textual and visual data. The core architecture of these models consists of a visual encoder, a text encoder, and a cross-modal fusion module. The visual encoder (such as CLIP-ViT or ResNet) is responsible for converting image or video frames into high-dimensional feature vectors, for example, extracting visual information such as texture and color from high-resolution images of specimens. The text encoder, based on the Transformer design, processes text input (such as "Name: Giant Panda Specimen") and generates semantic embeddings. The cross-modal fusion module aligns visual and textual features through a cross-attention mechanism, for example, associating "black on the shoulders and back" in an image with "giant panda fur color features" in the text description, and dynamically adjusts the weights of different modal information through a gating mechanism to ensure that the fused representation possesses both visual detail and semantic accuracy.

[0097] The pre-training phase aims to learn cross-modal semantic associations. Input data includes image-text pairs (e.g., images and descriptive text) and video clips. Through an image-text contrast learning task, the model learns to match positive samples (e.g., specimen images and correct descriptions) and reject negative samples (e.g., irrelevant image-text combinations). Simultaneously, the masking multimodal modeling task requires the model to jointly predict masked image regions and text terms; for example, masking "canine teeth" in an image and masking "carnivora" in text, forcing the model to infer missing content from context. This phase enables the model to acquire cross-modal reasoning capabilities, such as inferring species classification from images or generating visual descriptions from text.

[0098] The fine-tuning phase optimizes the multimodal attribute completion task for natural artifacts. Input data includes textual attributes (e.g., "Name: Mamenchisaurus Fossil") and visual data (e.g., fossil photographs or videos of the collection process). The model learns to generate missing attributes (e.g., "Collection Stratum: Late Jurassic terrestrial red beds") or extended attributes (e.g., "Conservation Status: Slightly Weathered") through end-to-end training. Visual features are mapped to the textual semantic space; for example, "crack distribution" in an image is transformed into "Conservation Recommendation: Constant Humidity Environment." For video data, the model extracts temporal features (e.g., the operation flow of the collection tool) to support dynamic attribute inference (e.g., "Collection Time: 3 hours").

[0099] The model breaks through the limitations of a single modality. For example, by combining "bamboo leaf residue" in a specimen image with the text "name: giant panda," it infers "diet: bamboo," revealing implicit ecological relationships. The generated attributes are both professional and interpretable. For example, "climate adaptability: temperate monsoon" is accompanied by derived data such as "average annual temperature 15°C, precipitation 1200mm," supporting scientific research decisions.

[0100] In step S3, the knowledge completion system inputs the known attribute data of the target collection into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data.

[0101] The structured known attributes of the target collection are converted into natural language text and embedded with task instruction templates, such as "Please complete the following attributes: Name: Giant Panda subadult skull, Collection altitude: 2650 meters, Associated plants: Arrow bamboo → Collection location:", forming a text sequence that conforms to the model input format, so as to clearly guide the model to focus on the attribute completion task.

[0102] After parsing the input text, the knowledge completion model generates missing preset attributes (such as "collection location: Wolong National Nature Reserve, Sichuan") and extended attributes (such as "ecological adaptability: cold-resistant") based on known attributes. For example, given the input "Specimen name: Crested Ibis, protection level: national first-class", the model generates "endangered factors: habitat reduction, protection recommendation: wetland ecological restoration". The generated text results are parsed into structured fields through keyword extraction and rule matching, distinguishing between preset attributes (such as collection location) and extended attributes (such as protection recommendations), forming a standardized candidate attribute dataset.

[0103] In step S4, the knowledge completion system performs rule verification and expert review on the candidate attribute dataset to obtain supplementary attribute data for the target collection.

[0104] Based on a pre-defined rule base, the system performs logical conflict detection on candidate attribute datasets. For example, it verifies whether the "collection time" is later than the specimen's entry time, or checks whether the "protection level" meets the standards of the International Union for Conservation of Nature (IUCN). If a conflict is found, such as "polar species-collection site-tropical," the system automatically marks it as conflicting data. Simultaneously, it calls external authoritative database interfaces (such as GBIF species distribution data and WorldClim climate data) for scientific verification, for example, comparing the distribution range of "giant panda specimen-collection site" with GBIF records to eliminate unreasonable results. This rule base can be dynamically optimized through automatic discovery of verification rules, reducing the workload of manually defining rules.

[0105] According to one example, attribute data that passes rule validation enters the automatic data entry process (i.e., proceeds to step S5), while conflicting or low-confidence results are pushed to the expert review platform. The review platform provides an interactive interface that displays candidate attributes, conflict details, and reference evidence (such as related literature and similar specimen data). Combined with visual evidence (such as species distribution maps and climate heat maps), it assists human decision-making. Experts can correct erroneous attributes (such as changing "collection location: Sahara Desert" to "Wolong, Sichuan") or confirm extended attributes (such as "storage recommendation: humidity ≤50%)".

[0106] In step S5, the knowledge completion system inserts the supplementary attribute data of the target collection into its knowledge graph and records the source of the completion.

[0107] According to one example, the expert review results are synchronized to the knowledge graph in real time, recording the correction trajectory and reviewer information to ensure data traceability.

[0108] The approved supplementary attribute data is integrated with the original data to dynamically update the knowledge graph nodes of the target collection.

[0109] In some embodiments, supplementary attribute data triggers incremental model learning, optimizing subsequent completion accuracy. For example, the supplementary attribute data is fused with the original data and added to the training dataset as sample data of the target collection for subsequent model training. Through periodic fine-tuning of the model, an intelligent closed-loop optimization process of "model self-diagnosis - expert correction - targeted optimization" is achieved.

[0110] It should be noted that the embodiments of this disclosure can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the embodiments of this disclosure can be executed by a processor to implement the steps or functions described above. Similarly, the software program (including associated data structures) of the embodiments of this disclosure can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of the embodiments of this disclosure can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0111] Furthermore, at least a portion of the embodiments of this disclosure can be applied as computer program products, such as computer program instructions, which, when executed by a computing device, can invoke or provide methods and / or technical solutions according to the embodiments of this disclosure through the operation of the computing device. The program instructions that invoke / provide the methods of the embodiments of this disclosure may be stored in a fixed or removable recording medium, and / or transmitted via a data stream in a broadcast or other signal carrying medium, and / or stored in the working memory of a computing device operating according to the program instructions.

[0112] It will be apparent to those skilled in the art that the embodiments of this disclosure are not limited to the details of the exemplary embodiments described above, and that the embodiments of this disclosure can be implemented in other specific forms without departing from the spirit or essential characteristics of the embodiments of this disclosure. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the embodiments of this disclosure is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be encompassed within the embodiments of this disclosure. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is apparent that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the system claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.

Claims

1. A method for knowledge completion in natural collections, wherein, The method includes the following steps: A training dataset for a knowledge completion model is constructed based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection. The knowledge completion model is trained based on the training dataset, wherein the training tasks include preset attribute completion and extended attribute generation; The known attribute data of the target collection is input into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data. The candidate attribute dataset is subjected to rule validation and expert review to obtain supplementary attribute data for the target collection. The supplementary attribute data is inserted into the knowledge graph of the target collection, and the source of the supplement is recorded.

2. The method according to claim 1, wherein, The knowledge completion model includes a large textual language model. The multimodal data includes image data and / or video data of the collection, and textual description information of the image data and / or video data is generated, which together with other attribute data of the collection constitutes the attribute data set of the collection.

3. The method according to claim 1, wherein, The knowledge completion model includes a multimodal large language model. The multimodal data includes image data and / or video data of the collection. Feature information of the image data and / or video data is extracted and combined with other attribute data of the collection to form the attribute data set of the collection.

4. The method according to claim 1, wherein, The knowledge completion model adopts a cascaded structure, including a media description generation model and a text-based large language model. The multimodal data includes image data and / or video data of the collection. The media description generation model generates text description information for the image data and / or video data, which, together with other attribute data of the collection, constitutes the attribute data set of the collection. This set is then input into the text-based large language model to jointly train the media description generation model and the text-based large language model.

5. The method according to claim 1, wherein, During the training of the knowledge completion model, the base parameters of the knowledge completion model are frozen, and only the low-rank matrix is ​​trained.

6. The method according to claim 1, wherein, The rule verification and expert review steps specifically include: Perform rule validation on the candidate attribute dataset; The candidate attribute data that passes the verification will be pushed to the expert review platform to receive feedback for manual correction.

7. The method according to claim 5, wherein, The method also includes the following steps: The approved supplementary attribute data is merged with the known attribute data of the target collection and then added to the training dataset as the attribute dataset of the target collection.

8. A knowledge completion device for natural collections, wherein, The device includes a memory and a processor. The memory stores computer program instructions, which, when executed by the processor, configure the device to perform the following operations: A training dataset for a knowledge completion model is constructed based on multimodal data from the natural history museum field, where each training sample is a set of attribute data for a natural history collection. The knowledge completion model is trained based on the training dataset, wherein the training tasks include preset attribute completion and extended attribute generation; The known attribute data of the target collection is input into the trained knowledge completion model to generate a candidate attribute dataset of the target collection, including missing preset attribute data and extended attribute data. The candidate attribute dataset is subjected to rule validation and expert review to obtain supplementary attribute data for the target collection. The supplementary attribute data is inserted into the knowledge graph of the target collection, and the source of the supplement is recorded.

9. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed by a computer device, the computer device is configured to perform the method as described in any one of claims 1 to 7.

10. A computer program storage medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a computer device, the computer device is configured to perform the method as described in any one of claims 1 to 7.

Images