An Automated Asset Labeling and Indexing Method Based on Multi-Source Feature Fusion
By using multi-source feature fusion and closed-loop optimization mechanisms, the consistency and efficiency issues in image asset annotation and retrieval index generation are resolved, achieving efficient and accurate annotation and retrieval while reducing system complexity.
Patent Information
- Application Number
- CN202611237409.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies for image asset annotation and retrieval index generation suffer from high costs, low efficiency, difficulty in ensuring consistency, and lack of multimodal fusion and self-evaluation capabilities, making it impossible to effectively filter and correct erroneous annotations.
A multi-source feature fusion method is adopted, which extracts metadata, visual semantics and optical character recognition features, combines modality-to-label dependency matrix to dynamically adjust weights, generates semantic labels and calculates consistency scores, uses random hash function to generate binary hash code to build index, supports approximate semantic retrieval, and automatically filters and corrects labels through closed-loop optimization mechanism.
It improves the accuracy and efficiency of annotation, reduces system complexity, enables automatic evaluation and optimization of annotation quality, and supports efficient multimodal fusion and approximate semantic retrieval.
Smart Images

Figure CN123115134A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing and fusion technology, specifically to a method for automated asset labeling and index generation based on multi-source feature fusion. Background Technology
[0002] As enterprises deepen their digital transformation, the number of digital assets is exploding. Image assets such as product photos, scanned contracts, and design drawings need to be organized and retrieved efficiently. Traditional asset labeling relies on manual operation, which is costly, inefficient, and difficult to guarantee consistency. Existing automated labeling methods usually rely on a single information source, either performing only image recognition and ignoring embedded text in the image, or performing only optical character recognition and ignoring visual semantic content. Even multimodal fusion methods mostly use fixed-weighted averages, without considering the different degrees of dependence of different labels on different modalities.
[0003] Regarding retrieval indexing, traditional methods using inverted indexes only support exact keyword matching and cannot rank similarity based on tag weights; while vector indexes require specialized database plugins, increasing system complexity. Existing systems lack automatic evaluation and closed-loop optimization mechanisms for annotation quality, making it impossible to effectively filter and correct potentially erroneous annotations.
[0004] Therefore, there is a need for an automated annotation and index generation method that can integrate multi-source features, dynamically adjust modality weights, generate compact indexes, and has self-evaluation capabilities. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing an automated asset labeling and index generation method based on multi-source feature fusion.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] An automated asset annotation and index generation method based on multi-source feature fusion includes the following steps:
[0008] Step S1: Capture the asynchronous asset upload event generated by the object storage service, and publish the asynchronous event to the message queue to asynchronously trigger the distributed feature computing cluster;
[0009] Step S2: Extract metadata features, visual semantic features, and optical character recognition text features from the uploaded image assets;
[0010] Step S3: Based on the visual semantic features and the optical character recognition text features, combined with the pre-calculated modality-to-label dependency matrix, dynamically fuse and generate semantic labels for the assets. At the same time, generate attribute labels based on the metadata features and heuristic rules, and calculate the consistency score between visual and optical character recognition. Mark assets with consistency scores below the threshold as requiring manual review.
[0011] Step S4: Construct a tag weight vector from the semantic tags of each asset and the fusion confidence of each semantic tag. Map the tag weight vector to a fixed-length binary hash code using a set of fixed random hash functions. Store the binary hash code in the database and build a B-tree index. At the same time, build an inverted index for the semantic tag field.
[0012] Step S5: Periodically scan assets marked as requiring manual review for administrator approval and correction, and recalculate the modality-to-label dependency matrix based on the corrected data.
[0013] Further, in step S2, the metadata features include file size, creation time, and security level; the visual semantic features are the label confidence vectors output by a random forest classifier after extracting image features through a convolutional neural network with fixed weights; the optical character recognition text features are the continuous confidence vectors obtained by extracting text boxes in the image through an optical character recognition algorithm, calculating the weighted importance score of each keyword, and then taking the maximum normalized weighted importance score in the keyword set for each label.
[0014] Furthermore, in step S2, extracting optical character recognition text features specifically includes the following steps:
[0015] The system has a pre-built asset thesaurus, which configures a set of keywords for each predefined tag;
[0016] Run the optical character recognition algorithm to detect all text boxes in the image, record the area, height, and center point coordinates of each text box, and calculate the total area of the image, the maximum height of all text boxes in the image, and the system's preset hotspot coordinates;
[0017] For each word extracted from the optical character recognition results, if it belongs to a keyword of a certain tag in the asset thesaurus, a weighted importance score is calculated.
[0018] The top ten words with the highest weighted importance scores are selected as optical character recognition keywords. For each label, the weighted importance scores of all words in the keyword set of that label are calculated. The maximum value of the weighted importance scores is divided by the maximum weighted importance scores of all words in the current image to obtain the continuous confidence score of the optical character recognition for that label, thereby generating the continuous confidence score vector of the optical character recognition label.
[0019] Furthermore, the weighted importance score is equal to the sum of the contributions of all text boxes containing the word; the contribution of each text box is obtained by multiplying three factors: the first factor is an indicator function, which is 1 when the text box contains the word and 0 otherwise; the second factor is a linear weighting of area and font size, that is, the area weight coefficient multiplied by the ratio of the text box area to the total area of the image, plus the font size weight coefficient multiplied by the ratio of the text box height to the height of the largest text box in the image; the third factor is a Gaussian decay factor, the exponent of which is the square of the Euclidean distance from the center point of the text box to the system's preset hotspot coordinates divided by the square of twice the Gaussian bandwidth parameter.
[0020] Further, in step S3, the modality-to-label dependency matrix is calculated using an offline validation set: for each label, the F1 score predicted using only the visual classifier and the F1 score predicted using only optical character recognition keyword matching are calculated respectively; the visual dependency weight is equal to the visual F1 score divided by the sum of the visual F1 score and the optical character recognition F1 score; the optical character recognition dependency weight is equal to the optical character recognition F1 score divided by the sum of the visual F1 score and the optical character recognition F1 score; the sum of the visual dependency weight and the optical character recognition dependency weight is 1.
[0021] Further, in step S3, the specific method of dynamically fusing and generating semantic labels is as follows: for each label, the visual dependency weight of the label is multiplied by the confidence score output by the visual classifier, and the optical character recognition dependency weight of the label is multiplied by the component corresponding to the label in the continuous confidence score vector of optical character recognition to obtain the fusion confidence score of the label; labels with a fusion confidence score greater than 0.5 are selected as the semantic labels of the asset.
[0022] Further, in step S3, the formula for calculating the consistency score is: the consistency score equals 1 minus the average absolute difference of all labels; the absolute difference of each label is the absolute value of the difference between the confidence score output by the visual classifier and the corresponding component in the continuous confidence vector of optical character recognition; when the consistency score is lower than 0.4, the asset is marked as requiring manual review.
[0023] Further, in step S4, the fixed random hash function is generated in the following way: the hash bit length is selected as 128 bits, and 128 random vectors with dimensions equal to the total number of labels are randomly generated. Each component of each random vector is independently and identically distributed in a standard normal distribution. These 128 random vectors are generated once during system initialization and are permanently fixed.
[0024] Further, in step S4, the specific method of mapping the tag weight vector to the binary hash code is as follows: for each hash bit, calculate the dot product of the corresponding random vector and the tag weight vector. If the dot product is greater than or equal to 0, the hash bit is set to 1; otherwise, it is set to 0. Combine all 128 hash bits in order into a 16-byte binary string as the hash code of the asset.
[0025] Furthermore, step S5 specifically includes the following steps:
[0026] Step S5.1: The system background runs a scheduled task to scan the database daily for asset records that need to be manually verified and marked as true, and generates an audit report for the administrator to view;
[0027] Step S5.2: The administrator corrects the semantic tags of the assets based on the audit report;
[0028] Step S5.3: The system recalculates the fusion confidence of the corrected asset, updates the semantic label field and hash code field, and triggers automatic updates of the database index;
[0029] Step S5.4: Use the manually corrected labels as the new ground truth values, expand the validation set, recalculate the visual F1 score and optical character recognition F1 score for each label, and then update the visual dependency weights and optical character recognition dependency weights.
[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0031] 1. This invention calculates the weighted importance of optical character recognition words by using a Gaussian hotspot weighting function, comprehensively considering the area of the text box, font size, and spatial position, highlighting key text in the preset hotspot area of the image, thus solving the problem of traditional optical character recognition extracting all text without emphasis.
[0032] 2. This invention utilizes a modality-to-label dependency matrix to dynamically adjust the fusion weights based on the actual predictive capabilities of each label in visual and optical character recognition. This is more reasonable and accurate than a fixed weighted average, and is especially suitable for scenarios where different labels have significantly different degrees of dependence on different modalities.
[0033] 3. This invention compresses sparse label weight vectors into 128-bit binary hash codes and uses a database B-tree index to achieve approximate semantic retrieval without introducing an additional vector retrieval engine, thus reducing system complexity and deployment costs.
[0034] 4. This invention transforms metadata into attribute tags, enriching the searchable dimensions of assets; at the same time, it automatically filters assets with visual and optical character recognition conflicts through consistency scores, and after manual review, it feeds back to optimize the dependency matrix, forming a closed-loop improvement mechanism. Attached Figure Description
[0035] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0036] Figure 1 This is a flowchart illustrating an embodiment of the present invention;
[0037] Figure 2 This is a flowchart of the tag weight hash index and retrieval process according to an embodiment of the present invention. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] like Figure 1 As shown, the method for automated asset annotation and index generation based on multi-source feature fusion includes the following steps:
[0040] Step S1: Capture the asynchronous asset upload event generated by the object storage service, and publish the asynchronous event to the message queue to asynchronously trigger the distributed feature computing cluster;
[0041] Step S2: Extract metadata features, visual semantic features, and optical character recognition text features from the uploaded image assets;
[0042] Step S3: Based on the visual semantic features and the optical character recognition text features, combined with the pre-calculated modality-to-label dependency matrix, dynamically fuse and generate semantic labels for the assets. At the same time, generate attribute labels based on the metadata features and heuristic rules, and calculate the consistency score between visual and optical character recognition. Mark assets with consistency scores below the threshold as requiring manual review.
[0043] Step S4: Construct a tag weight vector from the semantic tags of each asset and the fusion confidence of each semantic tag. Map the tag weight vector to a fixed-length binary hash code using a set of fixed random hash functions. Store the binary hash code in the database and build a B-tree index. At the same time, build an inverted index for the semantic tag field.
[0044] Step S5: Periodically scan assets marked as requiring manual review for administrator approval and correction, and recalculate the modality-to-label dependency matrix based on the corrected data.
[0045] Users upload digital assets, such as product photos and scanned documents, to cloud object storage. The system captures the file creation event generated at the underlying level through an event listening mechanism, encapsulates the event into a message, and sends it to the message queue. The message queue distributes the task to idle distributed computing nodes in a first-in-first-out order. Users can close the page immediately after uploading, and the background asynchronously completes feature extraction and annotation without blocking user operations.
[0046] In step S2, the metadata features include file size, creation time, and security level; the visual semantic features are the label confidence vectors output by a random forest classifier after extracting image features through a convolutional neural network with fixed weights; the optical character recognition text features are the continuous confidence vectors obtained by extracting text boxes in the image through an optical character recognition algorithm, calculating the weighted importance score of each keyword, and then taking the maximum normalized weighted importance score in the keyword set for each label.
[0047] In step S2, extracting optical character recognition text features specifically includes the following steps:
[0048] The system has a pre-built asset thesaurus, which configures a set of keywords for each predefined tag;
[0049] Run the optical character recognition algorithm to detect all text boxes in the image, record the area, height, and center point coordinates of each text box, and calculate the total area of the image, the maximum height of all text boxes in the image, and the system's preset hotspot coordinates;
[0050] For each word extracted from the optical character recognition results, if it belongs to a keyword of a certain tag in the asset thesaurus, a weighted importance score is calculated.
[0051] The top ten words with the highest weighted importance scores are selected as optical character recognition keywords. For each label, the weighted importance scores of all words in the keyword set of that label are calculated. The maximum value of the weighted importance scores is divided by the maximum weighted importance scores of all words in the current image to obtain the continuous confidence score of the optical character recognition for that label, thereby generating the continuous confidence score vector of the optical character recognition label.
[0052] The weighted importance score is equal to the sum of the contributions of all text boxes containing the word. The contribution of each text box is obtained by multiplying three factors: the first factor is an indicator function, which is 1 when the text box contains the word and 0 otherwise; the second factor is a linear weighting of area and font size, that is, the area weight coefficient multiplied by the ratio of the text box area to the total area of the image, plus the font size weight coefficient multiplied by the ratio of the text box height to the height of the largest text box in the image; the third factor is a Gaussian decay factor, the exponent of which is the square of the Euclidean distance from the center point of the text box to the system's preset hotspot coordinates divided by the square of twice the Gaussian bandwidth parameter.
[0053] For each uploaded image asset, the system extracts features in parallel across the following three dimensions:
[0054] 1. Metadata dimension features: The following information is directly read from the file's metadata: file size, creation time, and security level, where the security level is specified by the user when uploading: L1 Public, L2 Internal, L3 Confidential, and L4 Top Secret.
[0055] 2. Visual semantic features: The image is processed using a pre-trained lightweight convolutional neural network MobileNetV2 with fixed weights, outputting a 1280-dimensional feature vector. An offline-trained random forest classifier is used as input to the above 1280-dimensional vector, and the output is a confidence vector of length N, where N represents the total number of predefined labels in the system. The sum of all components of the confidence vector is 1, and each component is the probability that the visual model believes the image content belongs to the i-th label. Labels with a probability greater than 0.6 are retained as visual candidates.
[0056] 3. Optical character recognition text dimension features: During system initialization, a set of keywords is configured for each predefined label. For example, the label "product" includes: {"product", "goods", "model", "Model"}, and the label "amount" includes: {"amount", "total", "total", "¥", "$"}. The word library is stored in a database table and can be manually supplemented at any time.
[0057] An optical character recognition (OCR) algorithm is applied to the image to generate K text boxes. Each text box k contains the following attributes: a list of all recognized words within the text box; the area of the text box; the height of the text box, reflecting the font size; and the coordinates of the center point of the text box. Simultaneously, global image attributes are calculated, including: the total area of the image, equal to the image width multiplied by its height; the maximum height of all text boxes in the current image; and system-preset hotspot coordinates. For most photos, the image center is used; for invoices, the lower right corner amount area can be used; and for contracts, the upper middle title area can be used. These hotspot coordinates are set once by the administrator based on the main asset type during system deployment. The same coordinate values are then used for all image assets of the same type, without recalculation or dynamic adjustment for each individual image. Normalized coordinates are used, representing the image width and height ratio. When calculating distances, the coordinates are multiplied by the image width and height to convert to absolute coordinates.
[0058] For any deduplicated word extracted from the optical character recognition results, check whether the word appears in the keyword set of a certain tag in the asset thesaurus. If not, ignore it; if it does, calculate its weighted importance using the following formula:
[0059]
[0060] in, The weighted importance score represents the word 't'. The higher the score, the more prominent the word is in the pre-defined hotspot region of the image, and the more likely it is to be used as a basis for annotation. This indicates an indicator function that takes a value of 1 if the k-th text box contains the word t, and 0 otherwise. Only text boxes containing t contribute to the summation. This represents the area weighting coefficient, which is empirically taken as 0.6. This represents the area of the k-th text box. Represents the total area of the image. This represents the font size weighting factor, which is empirically set to 0.4. This represents the height of the k-th text box. This indicates the maximum height of all text boxes in the current image. This represents the coordinates of the center point of the k-th text box. This represents the system's preset hotspot coordinates, i.e., the absolute pixel coordinates of the current image after hotspot coordinate transformation. This represents the square of the Euclidean distance from the center of the text box to the coordinates of the hotspot. The bandwidth parameter representing the Gaussian kernel is taken as 1 / 8 of the length of the image diagonal;
[0061] Calculate all candidate words The continuous confidence score for OCR is defined by the following formula:
[0062]
[0063] in, This represents the continuous confidence level of OCR for optical character recognition. Represents the maximum value of all words in the current image. value, This represents a very small constant, preventing division by zero.
[0064] In step S3, the modality-to-label dependency matrix is calculated using an offline validation set: for each label, the F1 score predicted using only the visual classifier and the F1 score predicted using only optical character recognition keyword matching are calculated; the visual dependency weight is equal to the visual F1 score divided by the sum of the visual F1 score and the optical character recognition F1 score; the optical character recognition dependency weight is equal to the optical character recognition F1 score divided by the sum of the visual F1 score and the optical character recognition F1 score; the sum of the visual dependency weight and the optical character recognition dependency weight is 1.
[0065] In step S3, the specific method for dynamically fusing and generating semantic labels is as follows: for each label, the visual dependency weight of the label is multiplied by the confidence score output by the visual classifier, and the optical character recognition dependency weight of the label is multiplied by the component corresponding to the label in the continuous confidence score vector of optical character recognition to obtain the fusion confidence score of the label; labels with a fusion confidence score greater than 0.5 are selected as the semantic labels of the asset.
[0066] In step S3, the formula for calculating the consistency score is: the consistency score equals 1 minus the average absolute difference of all labels; the absolute difference of each label is the absolute value of the difference between the confidence score output by the visual classifier and the corresponding component in the continuous confidence vector of optical character recognition; when the consistency score is lower than 0.4, the asset is marked as requiring manual review.
[0067] For each tag A visual classifier is used to make predictions on the validation set, and the F1 score of the label is calculated. The F1 score is the harmonic mean of precision and recall, and its value ranges from [0, 1].
[0068] Using optical character recognition keyword matching, if the optical character recognition keyword is matched, it is predicted as positive. Prediction is performed on the validation set, and the F1 score of the label is calculated.
[0069] Define the visual dependency weights and optical character recognition dependency weights separately, with the following formulas:
[0070]
[0071] in, Indicates label The degree of dependence on visual modality, This indicates the labels predicted using a visual classifier. F1 score, Indicates label The degree of dependence on optical character recognition modality, This indicates that the labels are predicted using optical character recognition keyword matching. F1 score, To represent a very small constant, preventing division by zero;
[0072] For new assets currently being processed, the final confidence level for each tag is calculated using a weighted sum, with the specific formula as follows:
[0073]
[0074] in, Indicates the fused tags Confidence level, This represents the probability output by the visual model. This represents the continuous confidence level of OCR for optical character recognition;
[0075] All confidence levels Tags with a value greater than 0.5 are used as semantic tags for the asset. These tag names are connected by commas and stored in the auto_tags field of the database.
[0076] Generate content-related attribute tags based on metadata features and simple heuristic rules:
[0077] If the file size is greater than 10 megabytes, a large file attribute label is generated; if it is between 1 megabyte and 10 megabytes, a medium file attribute label is generated; if it is less than 1 megabyte, a small file attribute label is generated.
[0078] Based on the hour of creation time, if it is between 0:00 and 6:00, a nighttime upload attribute label will be generated; if it is between 9:00 and 17:00, a working hour upload attribute label will be generated.
[0079] The security level, L1 / L2 / L3 / L4, is directly stored as an attribute tag.
[0080] The image is reduced to 100×100 pixels. The K-means algorithm is used to find the most frequent color cluster in the image. Then, the center color of the cluster is mapped to the closest one of the 11 primary colors to generate the main color label.
[0081] Apply Laplacian filtering to the image, calculate the variance of the filtering result. If the variance is greater than 1000, generate a high-resolution label; if it is between 300 and 1000, generate a medium-resolution label; and if it is less than 300, generate a low-resolution label.
[0082] Calculate the sum of the total areas of all optical character recognition text boxes, divide it by the total area of the image, and generate dense text labels if the ratio is greater than 0.3, generate text labels if the ratio is between 0.1 and 0.3, and generate no text labels if the ratio is less than 0.1.
[0083] The system makes a comprehensive judgment based on the file extension and the high-confidence tags in the visual tags. For example, if the file extension is .jpg and the visual tag contains the product, then a product photo tag is generated.
[0084] The aforementioned attribute tags are stored in the attr_tags field of the database in JSON format;
[0085] To determine whether the annotations from visual and optical character recognition sources are consistent, a consistency score is defined, with the specific formula as follows:
[0086]
[0087] in, Indicates the consistency score. Indicates the total number of tags;
[0088] If the consistency score is less than 0.4, the system marks the asset as requiring manual review and stores it in the need_review field of the database. Administrators can periodically view these assets and correct the labeling.
[0089] In step S4, the fixed random hash function is generated in the following way: the hash bit length is selected as 128 bits, and 128 random vectors with dimensions equal to the total number of labels are randomly generated. Each component of each random vector is independently and identically distributed in a standard normal distribution. These 128 random vectors are generated once during system initialization and are permanently fixed.
[0090] In step S4, the specific method for mapping the tag weight vector to a binary hash code is as follows: for each hash bit, calculate the dot product of the corresponding random vector and the tag weight vector. If the dot product is greater than or equal to 0, the hash bit is set to 1; otherwise, it is set to 0. Combine all 128 hash bits in order into a 16-byte binary string as the hash code of the asset.
[0091] To support subsequent rapid retrieval, this invention proposes a tag weight hash index, which compresses the tag weight vector of each asset into a fixed-length binary hash code and uses the B-tree index of the database to achieve efficient approximate matching.
[0092] For each asset a, define an N-dimensional vector, where the i-th component is the fusion confidence score calculated in step S3. ;
[0093] Choose a hash bit length of 128, randomly generate 128 N-dimensional vectors, and the components of each vector are independently and identically distributed according to the standard normal distribution. These random vectors are generated during system initialization and are permanently fixed, and do not change with assets.
[0094] For asset a, the h-th bit of the hash code is defined as:
[0095]
[0096] in, This represents the h-th hash bit of asset a, with a value of 0 or 1. Represents a symbolic function. This represents the i-th component of the h-th random vector. This represents the weight of asset a on label i;
[0097] Combine all 128 bits sequentially into a 16-byte binary string; store the binary string in the hash_code field of the database asset, with the field type being a fixed-length binary string of 16 bytes; create a B-tree index on the hash_code field; and create an inverted index on the auto_tags field to support traditional exact tag queries.
[0098] When a user submits a query, such as for an amount, the system first converts the query text into a query tag weight vector: The query text is segmented, and each query term is searched for a corresponding tag in the asset thesaurus. If a matching tag is found, a continuous weight value between 0 and 1 is assigned based on the degree of matching between the query term and the tag. If multiple query terms match the same tag, the highest weight value is used. Then, the generated weight vector is L2 normalized to ensure that the magnitude of the weight vector is within the same range as the magnitude of the asset tag weight vector. After this conversion, the same set of random hash functions used in the index building phase is used to calculate the hash code of the query according to the same hash calculation formula.
[0099] To find assets in the database that are similar to the query hash code, the system performs the following database operations:
[0100] The database's built-in Hamming distance function is used to calculate the difference in the number of bits between the hash_code field and the query hash code; the Hamming distance is limited to less than a preset threshold, such as 20, and all assets that meet the conditions are returned; these assets are the candidate set that are semantically similar to the query tag.
[0101] After feature extraction and label generation are completed for each asset, a hash code is immediately calculated and the data is written to the database. The database automatically maintains B-tree indexes and inverted indexes. If the administrator manually corrects the label of an asset later, the system will recalculate the fusion confidence and hash code of the asset and update the corresponding record in the database.
[0102] Step S5 specifically includes the following steps:
[0103] Step S5.1: The system background runs a scheduled task to scan the database daily for asset records that need to be manually verified and marked as true, and generates an audit report for the administrator to view;
[0104] Step S5.2: The administrator corrects the semantic tags of the assets based on the audit report;
[0105] Step S5.3: The system recalculates the fusion confidence of the corrected asset, updates the semantic label field and hash code field, and triggers automatic updates of the database index;
[0106] Step S5.4: Use the manually corrected labels as the new ground truth values, expand the validation set, recalculate the visual F1 score and optical character recognition F1 score for each label, and then update the visual dependency weights and optical character recognition dependency weights.
[0107] The system runs a scheduled task in the background to scan all asset records in the database where need_review = true and generate an audit report for the administrator to view.
[0108] Based on the audit report, the administrator corrects the semantic tags of assets individually or in batches: adding missing tags, deleting incorrect tags, and modifying tag names;
[0109] For each corrected asset, the system recalculates the fusion confidence score, recalculates the new hash code, and then updates the auto_tags and hash_code fields in the database. The database automatically updates the B-tree index and inverted index.
[0110] The system uses manually corrected labels from the most recent month as new ground truth values each month or according to a preset cycle to expand the original validation set. Then, it re-calculates the visual F1 score and optical character recognition F1 score, and updates the visual dependency weights and optical character recognition dependency weights. This allows it to continuously adapt to new asset distributions and improve fusion accuracy.
[0111] like Figure 2 The diagram illustrates the construction and retrieval process of the tag weight hash index:
[0112] 1. During the index building phase, for each asset that has been labeled, firstly, all semantic tags of the asset and their fusion confidence scores are collected to form a tag weight vector; then, a set of hash functions that are pre-generated randomly and permanently fixed during system initialization are used to perform hash calculations on the tag weight vector, and all hash bits are combined sequentially to form a 128-bit binary hash code; the hash code is stored in a dedicated field of the database, and a B-tree index is built on this field; at the same time, a traditional inverted index is built for the semantic tag field of the asset to support precise keyword queries.
[0113] 2. During the query and retrieval phase, the user-input query text is converted into a query tag weight vector: the query is segmented into words, and corresponding tags are matched in the asset thesaurus. A continuous weight value between 0 and 1 is assigned based on the degree of matching, and L2 normalization is performed. The same set of hash functions used in the index building phase is used to calculate the query's hash code. In the database, the Hamming distance function is used to calculate the number of different binary bits between the hash code of each asset record and the query hash code. Asset records with a Hamming distance less than a preset threshold are selected and returned as candidate results. An approximate search using a B-tree index based on the hash code is employed to quickly locate a set of assets semantically similar to the query without traversing all assets.
[0114] The examples described herein are merely preferred embodiments of the invention and are not intended to limit the concept and scope of the invention. Any modifications and improvements made by those skilled in the art to the technical solutions of the invention without departing from the design concept of the invention should fall within the protection scope of the invention.
Claims
1. A method for automated asset annotation and index generation based on multi-source feature fusion, characterized in that, Includes the following steps: Step S1: Capture the asynchronous asset upload event generated by the object storage service, and publish the asynchronous event to the message queue to asynchronously trigger the distributed feature computing cluster; Step S2: Extract metadata features, visual semantic features, and optical character recognition text features from the uploaded image assets; Step S3: Based on the visual semantic features and the optical character recognition text features, combined with the pre-calculated modality-to-label dependency matrix, dynamically fuse and generate semantic labels for the assets. At the same time, generate attribute labels based on the metadata features and heuristic rules, and calculate the consistency score between visual and optical character recognition. Mark assets with consistency scores below the threshold as requiring manual review. Step S4: Construct a tag weight vector from the semantic tags of each asset and the fusion confidence of each semantic tag. Map the tag weight vector to a fixed-length binary hash code using a set of fixed random hash functions. Store the binary hash code in the database and build a B-tree index. At the same time, build an inverted index for the semantic tag field. Step S5: Periodically scan assets marked as requiring manual review for administrator approval and correction, and recalculate the modality-to-label dependency matrix based on the corrected data.
2. The method according to claim 1, characterized in that, In step S2, the metadata features include file size, creation time, and security level; the visual semantic features are the label confidence vectors output by a random forest classifier after extracting image features through a convolutional neural network with fixed weights; the optical character recognition text features are the continuous confidence vectors obtained by extracting text boxes in the image through an optical character recognition algorithm, calculating the weighted importance score of each keyword, and then taking the maximum normalized weighted importance score in the keyword set for each label.
3. The method according to claim 2, characterized in that, In step S2, extracting optical character recognition text features specifically includes the following steps: The system has a pre-built asset thesaurus, which configures a set of keywords for each predefined tag; Run the optical character recognition algorithm to detect all text boxes in the image, record the area, height, and center point coordinates of each text box, and calculate the total area of the image, the maximum height of all text boxes in the image, and the system's preset hotspot coordinates; For each word extracted from the optical character recognition results, if it belongs to a keyword of a certain tag in the asset thesaurus, a weighted importance score is calculated. The top ten words with the highest weighted importance scores are selected as optical character recognition keywords. For each label, the weighted importance scores of all words in the keyword set are calculated, and the maximum value is divided by the maximum weighted importance score of all words in the current image to obtain the continuous confidence score of the optical character recognition for that label, thereby generating the continuous confidence score vector of the optical character recognition label.
4. The method according to claim 3, characterized in that, The weighted importance score is equal to the sum of the contributions of all text boxes containing the word. The contribution of each text box is obtained by multiplying three factors: the first factor is an indicator function, which is 1 when the text box contains the word and 0 otherwise; the second factor is a linear weighting of area and font size, that is, the area weight coefficient multiplied by the ratio of the text box area to the total area of the image, plus the font size weight coefficient multiplied by the ratio of the text box height to the height of the largest text box in the image; the third factor is a Gaussian decay factor, the exponent of which is the square of the Euclidean distance from the center point of the text box to the system's preset hotspot coordinates divided by the square of twice the Gaussian bandwidth parameter.
5. The method according to claim 4, characterized in that, In step S3, the modality-to-label dependency matrix is calculated using an offline validation set: for each label, the F1 score predicted using only the visual classifier and the F1 score predicted using only optical character recognition keyword matching are calculated; the visual dependency weight is equal to the visual F1 score divided by the sum of the visual F1 score and the optical character recognition F1 score; the optical character recognition dependency weight is equal to the optical character recognition F1 score divided by the sum of the visual F1 score and the optical character recognition F1 score. The sum of the visual dependency weight and the optical character recognition dependency weight is 1.
6. The method according to claim 5, characterized in that, In step S3, the specific method for dynamically fusing and generating semantic labels is as follows: for each label, the visual dependency weight of the label is multiplied by the confidence score output by the visual classifier, and the optical character recognition dependency weight of the label is multiplied by the component corresponding to the label in the continuous confidence score vector of optical character recognition to obtain the fusion confidence score of the label; labels with a fusion confidence score greater than 0.5 are selected as the semantic labels of the asset.
7. The method according to claim 6, characterized in that, In step S3, the formula for calculating the consistency score is: the consistency score equals 1 minus the average absolute difference of all labels; the absolute difference of each label is the absolute value of the difference between the confidence score output by the visual classifier and the corresponding component in the continuous confidence vector of optical character recognition; when the consistency score is lower than 0.4, the asset is marked as requiring manual review.
8. The method according to claim 7, characterized in that, In step S4, the fixed random hash function is generated in the following way: the hash bit length is selected as 128 bits, and 128 random vectors with dimensions equal to the total number of labels are randomly generated. Each component of each random vector is independently and identically distributed in a standard normal distribution. These 128 random vectors are generated once during system initialization and are permanently fixed.
9. The method according to claim 8, characterized in that, In step S4, the specific method for mapping the tag weight vector to a binary hash code is as follows: for each hash bit, calculate the dot product of the corresponding random vector and the tag weight vector. If the dot product is greater than or equal to 0, the hash bit is set to 1; otherwise, it is set to 0. Combine all 128 hash bits in order into a 16-byte binary string, which serves as the hash code of the asset.
10. The method according to claim 9, characterized in that, Step S5 specifically includes the following steps: Step S5.1: The system background runs a scheduled task to scan the database daily for asset records that need to be manually verified and marked as true, and generates an audit report for the administrator to view; Step S5.2: The administrator corrects the semantic tags of the assets based on the audit report; Step S5.3: The system recalculates the fusion confidence of the corrected asset, updates the semantic label field and hash code field, and triggers automatic updates of the database index; Step S5.4: Use the manually corrected labels as the new ground truth values, expand the validation set, recalculate the visual F1 score and optical character recognition F1 score for each label, and then update the visual dependency weights and optical character recognition dependency weights.
