Registration traceability method and system based on multi-modal feature association and storage medium

Abstract

The invention provides a registration traceability method and system based on multi-modal feature association and a storage medium, and relates to the technical field of information registration. The registration traceability method comprises the following steps: firstly, collecting single-mode data from different sources such as texts, images and images, and extracting corresponding feature vectors according to data types; and fusing the vectors to obtain multi-modal features, associating the multi-modal features with defined metadata, establishing a traceability knowledge base, and finally inputting user information to complete registration. The method has the beneficial effects that through multi-source data acquisition, information one-sidedness of single data is avoided, and target features are comprehensively captured; the knowledge base established by binding the multi-modal features and the metadata enables the registration information to have both original features and structured semantic information, provides rich query dimensions for traceability, and improves the integration efficiency of registration and traceability.

Description

Technical Field

[0001] This invention mainly relates to the field of information registration technology, specifically to a registration and traceability method, system, and storage medium based on multimodal feature association. Background Technology

[0002] Existing registration and traceability technologies mostly rely on single-modal data for information registration and tracing, which has the following limitations: single-modal data is difficult to fully characterize the features of the target object. For example, relying solely on text descriptions may lack visual details, while relying solely on images lacks semantic context, resulting in incomplete information during traceability; data from different sources are often stored independently, lacking effective association mechanisms, and cannot achieve cross-modal information complementarity and verification, thus reducing the accuracy and reliability of traceability; the knowledge base of traditional traceability systems is mostly based on simple structured data, lacking the integration of multi-dimensional features, and key data is easily tampered with, making it difficult to meet the requirements of high traceability scenarios.

[0003] Therefore, there is an urgent need for a registration and traceability method that can integrate multimodal data, establish effective associations, and build a reliable knowledge base to solve the problems of incomplete information and insufficient associations in existing technologies. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a registration and tracing method based on multimodal feature association, which addresses the shortcomings of the existing technology.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A registration and tracing method based on multimodal feature association, comprising the following steps: Collect multiple unimodal data from different sources, extract features from each unimodal data according to its data type, and obtain the unimodal feature vector corresponding to each unimodal data. All single-modal feature vectors are fused to obtain multimodal features; The multimodal features are associated with the defined metadata, and a traceability knowledge base is built based on the association relationship; User information is entered into the traceability knowledge base for registration.

[0006] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A registration and traceability system based on multimodal feature association, characterized in that it includes: The feature extraction module is used to collect multiple single-modal data from different sources, extract features from each single-modal data according to the data type of the single-modal data, and obtain the single-modal feature vector corresponding to each single-modal data. The feature fusion module is used to fuse all single-modal feature vectors to obtain multimodal features; The source tracing knowledge base construction module is used to associate the multimodal features with the defined metadata and build a source tracing knowledge base based on the association relationship; The registration module is used to input user information into the traceability knowledge base for registration.

[0007] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A registration and traceability system based on multimodal feature association includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the registration and traceability method based on multimodal feature association as described above. Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the registration and tracing method based on multimodal feature association as described above.

[0008] The beneficial effects of this invention are: by collecting multi-source single-modal data such as text, images, and pictures, it avoids the information bias caused by relying on a single data type and can capture the characteristics of the target object more comprehensively; By binding the fused multimodal features with metadata and building a knowledge base, the registration information not only contains the original data features, but also associates structured semantic information (such as identifiers, attributes, etc.), providing rich query dimensions for subsequent traceability. From data collection, feature extraction, and fusion to knowledge base construction and user registration, a complete technical process is formed to ensure that user information can be quickly incorporated into the system and support subsequent traceability queries, thereby improving the integrated efficiency of registration and traceability. Attached Figure Description

[0009] Figure 1 A flowchart of the registration and traceability method provided in the embodiments of the present invention; Figure 2 This is a block diagram of the registration and traceability system provided in an embodiment of the present invention. Detailed Implementation

[0010] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0011] Example 1: As Figure 1 As shown, this embodiment of the invention provides a registration and tracing method based on multimodal feature association, including the following steps: Collect multiple unimodal data from different sources, extract features from each unimodal data according to its data type, and obtain the unimodal feature vector corresponding to each unimodal data. All single-modal feature vectors are fused to obtain multimodal features; The multimodal features are associated with the defined metadata, and a traceability knowledge base is built based on the association relationship; User information is entered into the traceability knowledge base for registration.

[0012] In the above embodiments, by collecting multi-source single-modal data such as text, images, and pictures, the information bias caused by relying on a single data type is avoided, and the characteristics of the target object can be captured more comprehensively. By binding the fused multimodal features with metadata and building a knowledge base, the registration information not only contains the original data features, but also associates structured semantic information (such as identifiers, attributes, etc.), providing rich query dimensions for subsequent traceability. From data collection, feature extraction, and fusion to knowledge base construction and user registration, a complete technical process is formed to ensure that user information can be quickly incorporated into the system and support subsequent traceability queries, thereby improving the integrated efficiency of registration and traceability.

[0013] Preferably, multiple unimodal data from different sources are collected, and features are extracted from each unimodal data according to its data type to obtain the unimodal feature vector corresponding to each unimodal data, including: Collect unimodal data from different sources, including text data, image data, and picture data; For text data, a word embedding model is used to convert the text data into text feature vectors; Specifically, the text feature vector is represented as follows: , in, For words Word embedding vectors, The length of the text data.

[0014] For image data, visual feature vectors are extracted from the image data using spatiotemporal feature extraction methods; Specifically, the image data is a continuous frame sequence (containing dynamic visual information). Visual feature vectors are extracted by combining spatial features and temporal dimension information. The specific steps are as follows: S1.1 Image data preprocessing: Frame sampling is performed on the original image data (such as video), key frames are extracted at fixed time intervals (such as 25 frames per second), and redundant frames are removed to reduce the amount of computation. The sampled frame images are standardized by: unifying the size (e.g., scaling to 224×224 pixels), normalizing pixel values ​​(e.g., mapping to the [0,1] range), and removing salt-and-pepper noise (e.g., using median filtering).

[0015] S1.2 Spatial Feature Extraction: For each frame of the image, single-frame spatial features are extracted using a pre-trained convolutional neural network (such as ResNet-50): The convolutional layers of the network (such as conv1 to conv5) are used to capture local features such as edges, textures, and object contours within the frame; The output of the network's global average pooling layer is taken as the spatial feature vector of a single frame, with dimension [missing information]. (where k is the frame index).

[0016] S1.3, Temporal Feature Fusion: The spatial feature vector sequence of consecutive frames (n is the total number of frames) Input to a bidirectional long short-term memory network (Bi-LSTM): Forward LSTM captures the temporal dependencies from frame 1 to frame n, and backward LSTM captures the inverse dependencies from frame n to frame 1. Take the hidden state of the last time step of the Bi-LSTM as the global temporal feature vector. It integrates information on motion trends between frames (such as object displacement and attitude changes).

[0017] S1.4, Visual Feature Vector Generation: Spatial and temporal features are fused: Weights are assigned to single-frame spatial features using an attention mechanism (e.g., dynamically highlighting the feature weights of frames with intense motion), as shown in the formula: ,in The attention weights for the spatial features of the k-th frame (satisfying) ), For the fusion weights of time features (e.g.) = 0.3), ultimately generating visual feature vectors from the image data. (D is the target dimension, such as 1024).

[0018] For image data, image feature vectors are extracted from the image data using a pre-trained ResNet model.

[0019] The specific steps are as follows: S2.1 Image Preprocessing: The original images are uniformly adjusted to the input size adapted to the ResNet model (e.g., 224×224 pixels), and the pixel values ​​are normalized according to the standards used during model pre-training (e.g., based on the mean and standard deviation of the ImageNet dataset) to eliminate the influence of lighting and size differences on feature extraction.

[0020] S2.2 Feature Extraction: The preprocessed image is input into the pre-trained ResNet model. The model's convolutional layers (such as conv1 to conv5_x) are used to extract local details (edges, textures) and global semantic (object structure) features of the image step by step. Finally, the high-dimensional feature map is transformed into a fixed-dimensional vector through a global average pooling layer (such as ResNet-50 outputting a 2048-dimensional feature vector).

[0021] S2.3, Feature Output: The vector output by the pooling layer is directly used as the image feature vector. This vector retains the key visual information of the image and can be directly used for subsequent fusion processing with modal features such as text and images.

[0022] In the above embodiments, a differentiated extraction method is adopted for the characteristics of text, image, and image data—text retains semantic information through word embedding, images are extracted through spatiotemporal feature extraction to take into account both spatial details and temporal dynamics, and images retain key visual features through ResNet to ensure that the core information of each modality is not lost; Image data is preprocessed by frame sampling, standardization, and denoising, and image size normalization to reduce redundant information and noise interference. Meanwhile, image feature extraction combines convolutional neural networks (spatial) and Bi-LSTM (temporal), and image features utilize pre-trained models to improve the accuracy and robustness of features. This lays the foundation for subsequent fusion: Through a structured extraction process, each single-modal feature vector has a unified format and clear physical meaning, which reduces the difficulty of subsequent cross-modal fusion and ensures the effectiveness of the fusion process.

[0023] Preferably, all single-modal feature vectors are fused to obtain multimodal features, including: A linear transformation mapping network is used to project text feature vectors, visual feature vectors, and image feature vectors into a shared space for feature alignment. The linear transformation mapping network is as follows: , , , in, These are text feature vectors, visual feature vectors, and image feature vectors. The aligned text feature vector, aligned visual feature vector, and aligned image feature vector are given. These are text modality mapping matrices, visual modality mapping matrices, and image modality mapping matrices. These are the text modality mapping bias vector, the visual modality mapping bias vector, and the image modality mapping bias vector; The aligned text feature vector, aligned visual feature vector, and aligned image feature vector are weighted and fused using a weighted fusion formula to obtain multimodal fused features. The weighted fusion formula is as follows: , in, These are the weights of the aligned text feature vector, aligned visual feature vector, and aligned image feature vector, respectively.

[0024] In the above embodiments, text, visual, and image features are projected onto a shared space through a linear transformation mapping network, eliminating the differences in dimension and distribution of features of different modalities, realizing the unification of the feature space, and providing a prerequisite for cross-modal fusion; By fusing aligned features using a weighted formula, weights can be dynamically allocated according to the importance of different modalities (e.g., image features have higher weights in dynamic scenes), enabling the fused multimodal features to more accurately reflect the core information of the original data and improve the expressive power of the features. The fusion process takes into account both the uniqueness and relevance of each modality, so that the final multimodal features retain the key information of the single modality and have cross-modal consistency, providing a high-quality feature foundation for subsequent association with metadata and traceability queries.

[0025] Preferably, the multimodal features are associated with defined metadata, and a traceability knowledge base is built based on the association relationship, including: Metadata is defined based on the requirements of the traceability scenario; A positive association from multimodal features to metadata and a negative association from metadata to multimodal features are established based on the hash mapping method. A traceability knowledge base is constructed, comprising a storage layer, an index layer, and an interface layer. The storage layer uses a distributed database to store multimodal feature vectors, metadata, and their forward and reverse associations, and uses a blockchain module to perform chained storage of key data in the metadata. The index layer is used to construct high-dimensional indexes for the multimodal feature vectors and tree indexes for the metadata. The interface layer is used to establish verification events that associate user registration information with metadata.

[0026] Preferably, establishing a positive association from multimodal features to metadata and a reverse association from metadata to multimodal features based on a hash mapping method includes: Perform hash operations on multimodal features to generate unique multimodal feature hash values, and bind the multimodal feature hash values ​​with the corresponding metadata to form key-value pairs with a positive association between the multimodal feature hash values ​​and the metadata; Extract key identification information from the metadata, perform hash operation on the key identification information to generate a unique metadata hash value, and bind the metadata hash value with the multimodal feature to form a key-value pair with a reverse association between the metadata hash value and the multimodal feature.

[0027] Specifically, metadata is defined based on the needs of the traceability scenario, including: Metadata Classification: Based on the specific traceability scenario (such as logistics management, document registration, and goods circulation), metadata includes two core types of information: Basic Identifier Class: Used to uniquely identify entities or events, including item ID, event number, registration timestamp (accurate to milliseconds), registration location (latitude and longitude or location code), operator ID, etc., to ensure the uniqueness of the traceability chain; Main attribute class: describes the core characteristics of an entity or event, such as the weight, material, and destination of an item in a logistics scenario; the type, validity period, and holder information of a document in document registration. Attribute fields can be dynamically added or removed according to the scenario.

[0028] Specifically, positive and negative associations are established based on hash mapping methods: S3.1, Positive Association (Multimodal Features → Metadata): The fused multimodal features are hashed (using the SHA-256 algorithm) to generate a fixed-length (256-bit) multimodal feature hash value, which serves as the unique identifier of the feature. The hash value is bound to the corresponding metadata (including basic identifier class and main attribute class) to form a key-value pair "multimodal feature hash value → complete metadata information", ensuring that the metadata can be directly located through the feature hash.

[0029] S3.2, Reverse Association (Metadata → Multimodal Features): Extract key information of the basic identifier class from the metadata (such as item ID + timestamp, the combination of the two ensures uniqueness), and perform a hash operation on it to generate a metadata hash value; The metadata hash value is bound to the original multimodal features (including unimodal features and fused features of text, images, and images) to form a key-value pair "metadata hash value → multimodal feature set", which supports tracing back the original features through metadata.

[0030] S3.3 Association Validation: After establishing the association, cross-validation is used to ensure consistency (e.g., the metadata hash value of the metadata queried using the hash value of the multimodal feature must be able to match the original multimodal feature in reverse). If there is inconsistency, the data correction process is triggered.

[0031] Building a knowledge base for tracing origins includes: S4.1 Storage Layer Design: Use a distributed database (such as HBase) to store multimodal feature vectors, metadata, and forward and reverse association key-value pairs, and leverage the distributed architecture to support high-concurrency read and write operations (such as real-time writing and traceability querying of massive registration data). A blockchain module is introduced to store basic identification information (such as item ID and timestamp) in metadata in a chain: each block contains "block header (hash value of the previous block + timestamp of this block) + block body (basic identification metadata)", ensuring that key data is not tampered with through the immutability of blockchain and that the data modification history can be traced.

[0032] S4.2 Index Layer Design: Construct high-dimensional indexes for multimodal feature vectors (such as using the IVF-Flat index from the FAISS library), and accelerate the approximate nearest neighbor search of feature vectors through clustering and bucketing to improve cross-modal feature matching efficiency; Build tree indexes (such as B+ tree indexes) on metadata, using basic identifier information (such as item ID, timestamp) as index keys, to support fast and accurate metadata queries (such as retrieving the registration records of an item by time range).

[0033] S4.3 Interface Layer Design: Provide standardized interfaces (such as RESTful API) to support the input, query, and verification of user information: During registration, the interface layer triggers a validation event, comparing the user input information (such as the submitted item description) with the main attribute information in the metadata. If the matching degree is greater than or equal to a preset threshold (such as 85%), registration is allowed and the knowledge base is updated; if the matching fails, an exception warning is triggered (such as prompting "Information is inconsistent, please check"). It supports a traceability query interface, allowing users to initiate queries using multimodal features (such as uploaded images) or metadata (such as item IDs). The interface layer calls the index layer to quickly return related results.

[0034] S4.4 Metadata Standardization: Standardize the format of various metadata (e.g., timestamps use ISO 8601 format, IDs use "prefix + number sequence" structure) to avoid association failure due to format confusion.

[0035] In the above embodiments, a forward (feature → metadata) and reverse (metadata → feature) association between multimodal features and metadata is established through hash mapping, which ensures the unique identification of data and bidirectional traceability. It can locate metadata through features and also look up the original features through metadata, thus improving the flexibility of traceability. The knowledge base features a multi-layered design that balances efficiency and security. The storage layer uses a distributed database to support high-concurrency read and write operations, and combines blockchain to store key metadata (such as identifiers and timestamps) in a chain. The immutability of blockchain ensures that core data is not tampered with, while also supporting the tracing of modification history. The index layer significantly improves query efficiency for massive amounts of data through high-dimensional indexes (accelerating feature matching) and tree indexes (accelerating metadata queries); The interface layer's verification mechanism ensures the consistency between user registration information and metadata, reducing the risk of erroneous data entry; Metadata standardization enhances system compatibility: Based on the traceability scenario, metadata (basic identifier class + main attribute class) is defined and its format is standardized to ensure that data from different sources and types can be uniformly incorporated into the knowledge base, thereby enhancing the system's universality and scalability.

[0036] Example 2: Figure 2 As shown, this embodiment of the invention also provides a registration and tracing system based on multimodal feature association, including: The feature extraction module is used to collect multiple single-modal data from different sources, extract features from each single-modal data according to the data type of the single-modal data, and obtain the single-modal feature vector corresponding to each single-modal data. The feature fusion module is used to fuse all single-modal feature vectors to obtain multimodal features; The source tracing knowledge base construction module is used to associate the multimodal features with the defined metadata and build a source tracing knowledge base based on the association relationship; The registration module is used to input user information into the traceability knowledge base for registration.

[0037] Preferably, multiple unimodal data from different sources are collected, and features are extracted from each unimodal data according to its data type to obtain the unimodal feature vector corresponding to each unimodal data, including: Collect unimodal data from different sources, including text data, image data, and picture data; For text data, a word embedding model is used to convert the text data into text feature vectors; For image data, visual feature vectors are extracted from the image data using spatiotemporal feature extraction methods; For image data, image feature vectors are extracted from the image data using a pre-trained ResNet model.

[0038] Preferably, the multimodal features are associated with defined metadata to form a traceability knowledge base, including: Metadata is defined based on the requirements of the traceability scenario; A positive association from multimodal features to metadata and a negative association from metadata to multimodal features are established based on the hash mapping method. An initial traceability knowledge base is established, comprising a storage layer, an index layer, and an interface layer. The storage layer uses a distributed database to store multimodal feature vectors, metadata, and their forward and reverse associations, and uses a blockchain module to perform chained storage of key data in the metadata. The index layer is used to construct high-dimensional indexes for the multimodal feature vectors and tree indexes for the metadata. The interface layer is used to establish verification events that associate user registration information with metadata.

[0039] Example 2: This embodiment of the invention also provides a registration and traceability system based on multimodal feature association, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the registration and traceability method based on multimodal feature association as described above.

[0040] Example 3: This embodiment of the invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the registration and tracing method based on multimodal feature association as described above.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0042] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0043] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0044] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0045] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0046] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for registration provenance based on multi-modal feature association, characterized in that, The method comprises the following steps: Collecting multiple single-modal data of different sources, respectively extracting features of each single-modal data according to the data types of the single-modal data to obtain single-modal feature vectors corresponding to each single-modal data; Fusing all single-modal feature vectors to obtain multi-modal features; Associating the multi-modal features with defined metadata, and building a traceability knowledge base based on the association relationship; Inputting user information into the traceability knowledge base for registration.

2. The method of claim 1, wherein, Collecting multiple single-modal data of different sources, respectively extracting features of each single-modal data according to the data types of the single-modal data to obtain single-modal feature vectors corresponding to each single-modal data, comprising: Collecting single-modal data of different sources, wherein the single-modal data includes text data, image data, and image data; For text data, converting the text data into a text feature vector through a word embedding model; For image data, extracting a visual feature vector from the image data through a spatio-temporal feature extraction method; For image data, extracting an image feature vector from the image data through a pre-trained ResNet model.

3. The method of claim 2, wherein, Fusing all single-modal feature vectors to obtain multi-modal features, comprising: Projecting the text feature vector, the visual feature vector, and the image feature vector into a shared space for feature alignment processing through a linear transformation mapping network, wherein the linear transformation mapping network is: ， ， ， wherein, is a text feature vector, a visual feature vector, and an image feature vector, is an aligned text feature vector, an aligned visual feature vector, and an aligned image feature vector, is a text modality mapping matrix, a visual modality mapping matrix, and an image modality mapping matrix, is a text modality mapping bias vector, a visual modality mapping bias vector, and an image modality mapping bias vector; Performing weighted fusion processing on the aligned text feature vector, the aligned visual feature vector, and the aligned image feature vector through a weighted fusion formula to obtain multi-modal fusion features, wherein the weighted fusion formula is: ， wherein, are respectively weights of the aligned text feature vector, the aligned visual feature vector, the aligned image feature vector.

4. The method of claim 1, wherein, Associating the multi-modal features with defined metadata, and building a traceability knowledge base based on the association relationship, comprising: Defining metadata based on traceability scenario requirements; Establishing a forward association relationship from multi-modal features to metadata and a reverse association relationship from metadata to multi-modal features based on a hash mapping method; Building a traceability knowledge base, wherein the traceability knowledge base includes a storage layer, an index layer, and an interface layer; the storage layer is used to store multi-modal feature vectors, metadata, and their forward and reverse association relationships using a distributed database, and to store key data in the metadata in a chain based on a blockchain module; the index layer is used to construct a high-dimensional index for multi-modal feature vectors and a tree index for metadata; the interface layer is used to establish a verification event for associating user registration information with metadata.

5. The method of claim 4, wherein, Establishing a forward association relationship from multi-modal features to metadata and a reverse association relationship from metadata to multi-modal features based on a hash mapping method, comprising: Performing hash operation on the multi-modal features to generate a unique multi-modal feature hash value, binding the multi-modal feature hash value with the corresponding metadata to form a key-value pair of the multi-modal feature hash value and the forward association relationship with the metadata; Extracting key identification information from the metadata, performing hash operation on the key identification information to generate a unique metadata hash value, and binding the metadata hash value with the multi-modal features to form a key-value pair of the metadata hash value and the reverse association relationship with the multi-modal features.

6. A registration provenance system based on multi-modal feature association, characterized in that, Comprising: The feature extraction module is configured to collect multiple single-modal data of different sources, perform feature extraction on each single-modal data according to the data type of the single-modal data, and obtain a single-modal feature vector corresponding to each single-modal data. The feature fusion module is configured to perform fusion processing on all single-modal feature vectors to obtain multi-modal features. The provenance knowledge base building module is configured to associate the multi-modal features with defined metadata and build a provenance knowledge base based on the association relationship. The registration module is configured to input user information into the provenance knowledge base for registration.

7. The registry traceability system of claim 6, wherein, The feature extraction module is configured to collect multiple single-modal data of different sources, perform feature extraction on each single-modal data according to the data type of the single-modal data, and obtain a single-modal feature vector corresponding to each single-modal data. The feature extraction module is configured to collect multiple single-modal data of different sources, perform feature extraction on each single-modal data according to the data type of the single-modal data, and obtain a single-modal feature vector corresponding to each single-modal data. For text data, a word embedding model is used to convert the text data into a text feature vector. For image data, a pre-trained ResNet model is used to extract an image feature vector from the image data. The multi-modal features are associated with defined metadata to form a provenance knowledge base, including:

8. The registry traceability system of claim 7, wherein, Defining metadata based on provenance scenario requirements; Based on the hash mapping method, a forward association relationship from multi-modal features to metadata and a reverse association relationship from metadata to multi-modal features are established; An initial provenance knowledge base is built, which includes a storage layer, an index layer, and an interface layer; the storage layer is configured to store multi-modal feature vectors, metadata, and their forward and reverse association relationships using a distributed database, and to store key data in the metadata in a chain based on a blockchain module; the index layer is configured to construct a high-dimensional index for multi-modal feature vectors and a tree index for metadata; and the interface layer is configured to establish a verification event for associating user registration information with metadata. The computer program stored in the memory and executable on the processor, when the processor executes the computer program, implements the registration and provenance method based on multi-modal feature association according to any one of claims 1 to 5.

9. A registration provenance system based on multi-modal feature association, characterized in that, When the computer program is executed by the processor, the registration and provenance method based on multi-modal feature association according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. ​

Images