A lightweight iterative human-computer collaborative labeling method and system based on incremental learning and retrieval library dynamic updating

Abstract

The application discloses a kind of based on incremental learning and retrieval library dynamic updating light iterative man-machine collaborative labeling method and system.Freezing weight pre-training image encoder extracts initial normalized image features, adjusts feature space online through light learnable MLP adjuster, and adopts the nearest center-based retrieval library dynamic updating strategy, to realize the continuous self-adaptive optimization of labeling system.MLP adjuster adopts residual structure and weight all-zero initialization technology to ensure that the system does not decline in performance at the beginning of labeling, and can gradually activate and learn the feature offset of the vertical field as the accumulation of artificial confirmation samples, realizing the smooth transition from "zero disturbance" to "accurate adaptation".When adjuster training, triplet loss training is used to close similar features and push away dissimilar features, and an elastic weight solidification regularization constraint important parameter shift based on Fisher information matrix is introduced to effectively suppress catastrophic forgetting.

Description

Technical Field

[0001] This invention relates to the field of image data annotation, and in particular to a lightweight iterative human-computer collaborative annotation method and system based on incremental learning and dynamic updating of the retrieval library. Background Technology

[0002] With the widespread application of artificial intelligence technology in vertical industries, high-quality image annotation data has become an important foundation for the training and deployment of visual models. Existing image annotation tools typically support functions such as manual annotation, semi-automatic annotation, and result export, which can improve data processing efficiency to a certain extent.

[0003] However, traditional image annotation systems still have significant shortcomings in scenarios involving fine-grained classification such as food identification and continuously growing data. First, most existing annotation systems rely on manual image-by-image annotation, which is costly and inefficient. When the number of images is large, the number of categories is large, or the similarity between categories is high, the workload of manual annotation increases significantly.

[0004] Secondly, although some semi-automatic annotation systems incorporate pre-trained or classification models, these models are typically static. During use, the system cannot fully utilize the new samples that users continuously verify and correct, making it difficult to continuously optimize the automatic annotation capabilities as labeled data accumulates.

[0005] Furthermore, existing image classification methods based on incremental learning typically revolve around a complete classification network, such as expanding the classifier head, retraining the network, introducing knowledge distillation, or classifier weight constraints. When used in annotation systems, these methods often suffer from high training overhead, insufficiently lightweight online updates, and a lack of flexibility in adding new categories.

[0006] In existing incremental learning methods, task-based incremental learning requires task identifiers during testing, which is not suitable for open-label scenarios. Although class-based incremental learning does not require task identifiers and can handle the dynamic addition of new categories, the problem of catastrophic forgetting is more prominent. How to achieve lightweight and efficient feature space adaptation in class-based incremental scenarios is the core problem that this invention aims to solve.

[0007] Furthermore, traditional classification networks rely on fixed classifier heads to output class probabilities. When new classes continuously emerge, the classification layer structure needs modification or the model needs to be retrained, which is unfavorable for scenarios with dynamically changing classes in image annotation systems. Existing automatic annotation methods also typically lack effective control over low-confidence annotation results, indiscriminately using manually annotated and automatically annotated samples for model updates. This easily leads to erroneous automatic annotation results being directly written into the system, causing these incorrect automatically annotated samples to contaminate subsequent training data and creating an erroneous feedback loop.

[0008] Therefore, a lightweight incremental automatic annotation method is needed for image annotation systems, which can continuously optimize the system in the cycle of manual annotation, automatic annotation and manual correction, while avoiding frequent training of the complete classification network, thereby improving annotation efficiency and system sustainability. Summary of the Invention

[0009] The purpose of this invention is to address the problems of existing image annotation systems, such as difficulty in continuously improving automatic annotation capabilities in dynamic data scenarios, high cost of incremental training of complete models, inflexible expansion of new categories, and easy propagation of erroneous automatic annotation results. This invention proposes a lightweight iterative human-computer collaborative annotation method and system based on incremental learning and dynamic updating of the retrieval library.

[0010] The objective of this invention is achieved through the following technical solution: a lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database, the method comprising the following steps: Obtain manually labeled and unlabeled images; extract and normalize the features of the images using a pre-trained image encoder to obtain normalized image features; The normalized image features are input into the residual feature adjuster, and the output is the sum of the input image features and the feature offset as the adjusted image features. Based on the adjusted image features and category labels of manually confirmed and labeled images, a category feature set is constructed and the center of each category is calculated; according to the preset retrieval library update strategy, the feature sample set of each category is filtered or reorganized to form a category center retrieval library; For unlabeled images, perform the same feature extraction and adjustment process as manually confirmed labeled images to obtain the features to be labeled and adjusted; calculate the similarity between the features to be labeled and adjusted and each category center in the category center retrieval library, and take the category corresponding to the maximum similarity as the candidate automatic labeling category; write the labeling result when the maximum similarity meets the preset threshold condition, otherwise keep it in the state of waiting for manual labeling. Based on the confirmed or corrected samples, a residual feature adjuster is jointly trained using triplet loss and elastic weight solidification regularization loss, and the category center retrieval library is reconstructed according to the preset retrieval library update strategy to complete one round of incremental iterative update.

[0011] Furthermore, the residual feature adjuster includes at least two fully connected layers, and the residual feature adjuster is... ,in Represents the trainable parameters of the feature adjuster; The residual feature adjuster adjusts the normalized image features. Learned feature offsets: The adjusted image features can be represented as: in, Indicates the adjusted image features. This represents the feature offset learned by the residual MLP feature adjuster.

[0012] Furthermore, the residual feature adjuster includes at least two fully connected layers, and the weights and biases of all fully connected layers are initialized to zero; this ensures that the feature offset is always a zero vector before training begins, and the adjusted image features are exactly the same as the input normalized image features. In the initial state, the adjuster is equivalent to an identity mapping.

[0013] Furthermore, the category center retrieval library includes: a set of category labels, a set of sample features corresponding to each category, category center features for each category, a category center matrix, and a mapping relationship between category labels and category centers.

[0014] Furthermore, the preset retrieval database update strategy is a nearest neighbor center sampling strategy. For each category, the distance between the adjusted image features of each sample and the current category center is calculated. Priority is given to retaining several samples closest to the category center as the representative feature set for that category. The category center is obtained by averaging and normalizing multiple adjusted image features within the same category. The category center is obtained by calculating the mean of image features after adjusting for similar samples and then normalizing it.

[0015] Furthermore, the calculation process for the candidate automatically labeled categories includes: First, the original image features are extracted and normalized using a pre-trained image encoder. Then, the normalized image features are input into a residual feature adjuster to obtain the adjusted features to be labeled. For any category in the category-centered retrieval database Calculate the features to be annotated and adjusted. With Category Center Cosine similarity between them: in, Indicates unlabeled images and categories The similarity is used to determine the category with the highest similarity, and the category corresponding to the highest similarity is used as the candidate category for automatic labeling.

[0016] Furthermore, in the joint training of the residual feature adjuster using the triplet loss and the elastic weight solidification regularization loss, the loss function... ,in, The weight coefficients representing the parameter importance constraint terms. Represents the loss of the triplet. The loss represents the importance constraint of the representative parameter; The loss of the triplet is: in, Indicates the interval parameter. Represents the feature distance function; , , These represent the features of the anchor sample, the positive sample belonging to the same category as the anchor sample, and the negative sample belonging to a different category from the anchor sample, respectively, after being processed by the image encoder and the residual feature adjuster. The parameter importance constraint loss is: in, These are the optimal parameters for the residual MLP feature adjuster in the previous training phase. The parameters for the current training phase are: For the first Importance coefficients of each parameter The larger the value, the higher the value. The more important a parameter is to historical labeled samples or historical category centers, the stronger the constraint on that parameter will be in subsequent incremental training.

[0017] Furthermore, when reconstructing the category center retrieval database, the adjusted image features of manually confirmed and labeled images are preferentially selected as sample features in the category feature set.

[0018] According to another aspect of the specification, an image incremental annotation system based on a residual feature adjuster and a category center retrieval library for implementing the method is also provided. The system includes: an image data management module, a manual annotation module, an image feature extraction module, a residual feature adjustment module, a category center retrieval library maintenance module, an automatic annotation module, an incremental update module, a result export module, and an asynchronous task management module. The image data management module is used to import and manage image data; The manual annotation module is used to receive manual annotations, confirmations, or corrections from users regarding image categories; The image feature extraction module is used to extract normalized image features through a pre-trained image encoder; The residual feature adjustment module is used to generate adjusted image features through a residual feature adjuster; The category center retrieval library maintenance module is used to maintain the category feature set and category centers based on manually confirmed samples; The automatic annotation module is used to generate automatic annotation results based on the similarity between the adjusted image features of the image to be annotated and the category center; The incremental update module is used to update the residual feature adjuster and the category center retrieval library based on manually confirmed samples.

[0019] According to another aspect of the specification, an image incremental annotation device based on a residual feature adjuster and a category center retrieval library is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the image incremental annotation method based on a residual feature adjuster and a category center retrieval library.

[0020] The beneficial effects of this invention are: (1) Reduced model update cost. This invention does not require frequent training of the complete image classification network. Instead, it fixes or calls a pre-trained image encoder and trains only a lightweight residual MLP feature adjuster, which significantly reduces the computational cost of incremental updates. When ResNet50 ONNX is used as the encoder in the experiment, the feature extraction and inference efficiency is significantly better than ConvNeXt and CLIP.

[0021] (2) Improve the flexibility of new category expansion. This invention uses a category center retrieval library for category prediction, without relying on a fixed classification head. New categories only need to be added with manually confirmed samples and their category centers calculated to participate in automatic labeling. We verified the system's ability to handle large-scale fine-grained categories on a 568-category food dataset.

[0022] (3) Enhance the adaptability of the feature space and ensure that the initial performance does not degrade. The combination of residual structure and all-zero initialization ensures that the system can directly reuse the pre-trained feature space in the initial stage of labeling without performance degradation. As the samples accumulate, it smoothly transitions into a vertical domain feature adapter, achieving the incremental learning effect of 'no degradation at the start and continuous optimization through iteration'.

[0023] (4) Reduce the risk of catastrophic forgetting. By constraining the importance of parameters, excessive changes in historically important parameters can be limited, reducing the damage to the recognition ability of old categories caused by incremental training with new samples. Comparative experiments show that the EWC regularization method achieves the optimal balance between accuracy and storage overhead.

[0024] (5) Reduce the risk of automatic labeling propagation errors. By controlling the similarity threshold, the system only writes the automatic labeling results when the maximum similarity meets the threshold condition, and low-confidence samples are reserved for manual processing.

[0025] (6) Forming a continuous optimization closed loop. The system combines manual annotation, automatic annotation, manual correction and incremental updates, enabling the annotation system to be continuously optimized as it is used. After 20 rounds of human-machine collaborative iteration, the accuracy of automatic annotation has been continuously improved from the initial baseline to 95.67%, achieving a continuous optimization effect of 'the more it is labeled, the more accurate it becomes'. Attached Figure Description

[0026] Figure 1The diagram shows the overall architecture of the lightweight iterative human-computer collaborative annotation method and system based on incremental learning and dynamic updating of the retrieval library, as provided in this embodiment of the invention.

[0027] Figure 2 This is a flowchart of the image incremental annotation method provided in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the residual MLP feature adjuster structure provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram illustrating the dynamic updating of the category-centered retrieval database provided in an embodiment of the present invention.

[0030] Figure 5 This is a diagram showing the comparison of accuracy and inference time for ResNet50+MLP+EWC under different annotation ratios.

[0031] Figure 6 This is a schematic diagram of the image incremental annotation device provided in an embodiment of the present invention. Detailed Implementation

[0032] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 and Figure 2 As shown, this embodiment provides a lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database. This method can be applied to scenarios such as food image annotation.

[0034] (1) Image data import and manual confirmation of sample acquisition The system supports batch import of image data via ZIP archives. After uploading, it automatically parses the directory structure within the archive, creating corresponding folder records in the database according to the 'first-level directory / second-level directory / image file' structure, and storing the image files in a specified path on the server. Users can browse the folders in a tree structure through the front-end interface and manually annotate each image. The system records the 'manual_labeled' flag to distinguish between manually confirmed samples and automatically labeled samples, and uses this as a data source for subsequent feature adjuster training and category center retrieval library updates.

[0035] (2) Feature extraction from pre-trained image encoder The system uses a pre-trained image encoder to extract features from manually labeled images. The pre-trained image encoder can be ResNet, ConvNeXt, CLIP, Vision Transformer, or other pre-trained visual models. In one embodiment, the ResNet50 ONNX model is used as the image encoder. After the input image undergoes size normalization, color channel conversion, pixel normalization, and standardization, it is input to the encoder to obtain the image feature vector. Subsequently, L2 normalization is performed on the image feature vector. Let the input image be... The pre-trained image encoder is The original image features extracted by the pre-trained encoder can then be represented as: To reduce the impact of differences in the modulus of different image features on subsequent category center calculation and similarity retrieval, L2 normalization is performed on the original image features to obtain normalized image features: in, Represents normalized image features. The L2 norm representing the features of the original image. To prevent extremely small constants with a denominator of zero.

[0036] (3) Residual MLP feature adjustment like Figure 3 As shown, the residual MLP feature adjuster includes at least two fully connected layers. In a preferred embodiment, the residual MLP feature adjuster includes three fully connected layers. The pre-trained image encoder output dimension is 512. The first fully connected layer of the residual MLP feature adjuster maps the 512-dimensional input to a 256-dimensional hidden space, the second fully connected layer maintains the 256-dimensional dimension, and the third fully connected layer maps the 256-dimensional dimension back to the 512-dimensional dimension, thereby generating a feature offset of the same dimension as the input.

[0037] The residual MLP feature adjuster is used to learn feature offsets, and its output is the sum of the input image features and the feature offsets. That is: Adjusted image features = Input image features + Feature offsets. Let the residual MLP feature adjuster be... ,in These represent the trainable parameters of the feature adjuster. The residual MLP feature adjuster adjusts based on normalized image features. Learned feature offsets: Therefore, the adjusted image features can be represented as: in, Indicates the adjusted image features. This represents the feature offset learned by the residual MLP feature adjuster. The residual MLP feature adjuster consists of three fully connected layers, and its feature offset generation process can be represented as follows: in, These represent the weight matrices of the three fully connected layers. These represent the bias terms of the three fully connected layers, This represents a non-linear activation function, using the ReLU activation function.

[0038] In a preferred embodiment, the residual MLP feature adjuster initializes all weights and biases of fully connected layers to exactly zero during initialization. This zero-initialization design is the core technical means by which the system's performance is not degraded during the initial labeling phase: due to the residual structure, the adjuster output is the sum of the input features and the feature offsets; zero initialization ensures that the feature offsets are exactly zero before training begins, making the adjuster precisely equivalent to an identity mapping in the initial state, rather than approximately equivalent. This is fundamentally different from traditional random initialization adaptation networks—random initialization introduces uncontrollable feature perturbations when samples are insufficient, while zero initialization mathematically guarantees that the feature space in the initial state is completely consistent with the pre-trained encoder. During initialization, the weights and biases of each fully connected layer satisfy: Therefore, in the initial state, the feature offset output by the residual MLP feature adjuster is exactly zero: At this point, the adjusted image features are exactly equal to the normalized image features output by the pre-trained image encoder: .

[0039] Therefore, the residual structure and zero initialization together constitute the dual performance guarantee mechanism of this invention: zero initialization ensures that the feature offset starts from zero, and the system directly reuses the feature space of the pre-trained encoder without introducing any performance degradation; residual connection ensures that even if the adjuster parameters deviate in subsequent training, the feature offset is only a correction and superposition of the original features, and will not destroy the pre-trained features themselves. In the initial stage of annotation when the accumulation of manually confirmed samples is insufficient, this mechanism fundamentally prevents the collapse of the feature space and ensures the reliability of automatic annotation. As the number of manually confirmed samples continues to increase, the parameters of the fully connected layer are gradually optimized from zero by the triplet loss, and the feature offset gradually increases from zero to a direction suitable for the current vertical scene. The adjuster smoothly transitions from "exact identity mapping" to "vertical domain feature adapter", achieving an incremental learning effect of "no degradation at the beginning and continuous optimization through iteration".

[0040] (4) Construction of Category-Centered Retrieval Database like Figure 4As shown, the system constructs a category feature set based on the adjusted image features and category labels of manually confirmed samples. For each category, the system collects the adjusted image features of multiple samples within that category and calculates the category center. The category center can be obtained by averaging and normalizing the adjusted image features within the same category. For each category... Let its manually confirmed sample set be: in, Indicate category The number of samples manually confirmed in the middle Indicate category The Middle Sample images. After processing by a pre-trained image encoder and a residual MLP feature adjuster, the categories are... The Middle The adjusted image features corresponding to each sample are: .category The category center can be calculated from the mean of the image features after adjustment of samples of the same type: To facilitate subsequent cosine similarity calculation, the category centers are normalized: in, Indicate category The original category center, This represents the normalized category centers. The normalized category centers of all categories together constitute the category center retrieval database. ,in, This represents the set of categories in the current category-centered search database. This represents the category center retrieval database. If all category centers are arranged in rows, the category center matrix can be represented as: in, This indicates the number of categories in the category-centered search database.

[0041] The category center retrieval library includes a set of category labels, a set of sample features for each category, category center features for each category, a category center matrix, and the mapping relationship between category labels and category centers. Unlike traditional fixed classification heads, the category center retrieval library can dynamically expand with manually confirmed samples. When a new category appears, the system only needs to collect manually confirmed samples of that category and calculate the category centers to add the new category to the automatic labeling process.

[0042] (5) Automatic annotation of unannotated images For unlabeled images, the system first calls a pre-trained image encoder to extract normalized image features, and then inputs these features into a residual MLP feature adjuster to obtain the adjusted features to be labeled. Subsequently, the system calculates the similarity between the adjusted features to be labeled and the category centers in the category center retrieval database. In one embodiment, cosine similarity is used for unlabeled images. The system first extracts and normalizes the features of the original image using a pre-trained image encoder, resulting in: Then, the normalized image features are input into the residual MLP feature adjuster to obtain the adjusted features to be labeled: For any category in the category-centered retrieval database Calculate the features to be annotated and adjusted. With Category Center Cosine similarity between them: in, Indicates unlabeled images and categories The system uses the similarity score as the basis for automatic category labeling, selecting the category with the highest similarity score as the candidate category. The corresponding maximum similarity is: In one embodiment, a preset automatic labeling threshold is used. It can be set to 0.6. When the cosine similarity exceeds the preset automatic annotation threshold, it indicates that the adjusted features of the image to be annotated are highly consistent with the center direction of the category, and the system writes the candidate automatic annotation category into the annotation result of the image; when the maximum similarity does not exceed the preset threshold, it indicates that the features are not close enough to the centers of all known categories, the system does not write into the automatic annotation result, and keeps the image in a state awaiting manual annotation to avoid the propagation of incorrect annotations. This process can be represented as: in, Indicates unlabeled images The annotation results This indicates the preset automatic labeling threshold.

[0043] (6) Manual feedback and incremental updates Users confirm or correct the automatic annotation results. Confirmed or corrected samples are then added back to the manually confirmed sample set. The system updates the residual MLP feature adjuster based on the newly added manually confirmed samples and recalculates or incrementally updates the category center retrieval library, thus forming a closed loop of "manual annotation - train adjuster - update retrieval library - automatic annotation - manual correction - update again". When a user confirms or corrects a new sample... The category is The system then uses this sample as a new manually verified sample and calculates its adjusted image features: In one embodiment, the system reconstructs the category feature set and category centers based on all currently manually confirmed and labeled samples. In another embodiment, when the number of newly confirmed samples is small, the system can locally update the corresponding category centers using an incremental averaging method, without traversing all historical samples. The original number of manually confirmed samples was The original category center is Then, the class centers after adding new samples can be updated incrementally: The updated normalized category centers are: Using the above method, the system can incrementally update the category centers based on newly added manually confirmed samples without completely rebuilding the category center retrieval database each time, thereby improving the efficiency of the retrieval database update.

[0044] To improve the discriminative power of the adjusted image features, this embodiment constructs triplet training samples based on manually confirmed labeled images. Each triplet includes an anchor sample, a positive sample belonging to the same category as the anchor sample, and a negative sample belonging to a different category from the anchor sample.

[0045] The training objective is to narrow the distance between anchor samples and positive samples, and widen the distance between anchor samples and negative samples, making the adjusted feature space more suitable for the current annotation task. In one embodiment, a residual MLP feature adjuster is trained using triplet loss. For triplet training samples... ,in Indicates anchor point sample, This indicates a positive sample belonging to the same category as the anchor sample. This indicates a negative sample belonging to a different category from the anchor sample. After processing by the pre-trained image encoder and the residual MLP feature adjuster, the adjusted image features corresponding to the three are as follows: The triplet loss can be expressed as: in, Indicates the interval parameter. This represents the feature distance function. In one embodiment, the feature distance function is the Euclidean distance: The triplet loss improves the intra-class compactness and inter-class separability of the adjusted feature space by bringing the anchor sample closer to the positive sample and widening the distance between the anchor sample and the negative sample.

[0046] To reduce the disruption to existing category recognition capabilities during incremental updates, this embodiment can also introduce parameter importance constraints. The system saves the optimal parameters of the feature adjuster from the previous training phase and estimates the importance of each parameter. In the next training round, the loss function includes triplet loss and parameter importance constraint loss, which is used to limit excessive shifts in important parameters from previous training phases. Let the optimal parameters of the residual MLP feature adjuster in the previous training phase be... The parameters for the current training phase are: , No. The importance coefficients of the parameters are Then the parameter importance constraint loss can be expressed as: in, The larger the value, the higher the value. The more important a parameter is to historical labeled samples or historical class centers, the stronger the constraint on that parameter in subsequent incremental training. Therefore, the total training loss of the residual MLP feature adjuster can be expressed as: ,in, This represents the weighting coefficients of the parameter importance constraint term. Through this loss function, the system can learn the feature distribution of newly added manually confirmed samples while minimizing disruption to the existing class feature space.

[0047] Corresponding to the foregoing method embodiments, the present invention also provides an image incremental annotation system based on a residual feature adjuster and a category center retrieval library. This system includes an image data management module, a manual annotation module, an image feature extraction module, a residual feature adjustment module, a category center retrieval library maintenance module, an automatic annotation module, an incremental update module, a result export module, and an asynchronous task management module.

[0048] The image data management module is used to import, store, display, and manage image data; The manual annotation module is used to receive manual annotations, confirmations, or corrections from users regarding image categories; The image feature extraction module is used to call the pre-trained image encoder to preprocess and extract features from the input image, and generate normalized image features. The residual feature adjustment module is used to input normalized image features into the residual MLP feature adjuster to generate adjusted image features. The category center retrieval library maintenance module is used to maintain the category feature set based on manually confirmed samples, calculate the category center of each category, and save the category center matrix and category label mapping relationship. The automatic annotation module is used to perform feature extraction, feature adjustment, similarity retrieval and threshold judgment on unlabeled images, and write the automatic annotation results when the threshold conditions are met. The incremental update module is used to train a residual MLP feature adjuster based on newly added manually confirmed samples and to update the category center retrieval database. The result export module is used to export image annotation results for subsequent model training or business system use. The asynchronous task management module is used to put training and automatic annotation tasks into the background and return progress information to the front end, so as to avoid blocking user operations by batch feature extraction and retrieval library updates.

[0049] like Figure 5 The diagram shows a comparison of accuracy and inference time for ResNet50+MLP+EWC under different annotation ratios. The system combines manual annotation, automatic annotation, manual correction, and incremental updates, enabling continuous optimization as the annotation system is used. On a fine-grained food dataset containing 568 categories and over 10,000 images, after 20 rounds of human-machine collaborative iteration, the automatic annotation accuracy has continuously improved from the initial baseline to 95.67%, achieving a continuous optimization effect of 'the more it's labeled, the more accurate it becomes'.

[0050] Corresponding to the aforementioned embodiment of a lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database, the present invention also provides an embodiment of a lightweight iterative human-computer collaborative annotation device based on incremental learning and dynamic updating of the retrieval database.

[0051] See Figure 6 The present invention provides a lightweight iterative human-computer collaborative annotation device based on incremental learning and dynamic updating of the retrieval library, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval library as described in the above embodiment.

[0052] The embodiment of the lightweight iterative human-computer collaborative annotation device based on incremental learning and dynamic updating of the retrieval library provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is a lightweight iterative human-computer collaborative annotation device based on incremental learning and dynamic updating of the retrieval database provided by the present invention. (Except for...) Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0053] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0054] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0055] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval library as described in the above embodiments.

[0056] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0057] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval library.

[0058] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0059] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database, characterized in that, The method includes the following steps: Obtain manually labeled and unlabeled images; extract and normalize the features of the images using a pre-trained image encoder to obtain normalized image features; The normalized image features are input into the residual feature adjuster, and the output is the sum of the input image features and the feature offset as the adjusted image features. Based on the adjusted image features and category labels of manually confirmed and labeled images, a category feature set is constructed and the center of each category is calculated; according to the preset retrieval library update strategy, the feature sample set of each category is filtered or reorganized to form a category center retrieval library; For unlabeled images, perform the same feature extraction and adjustment process as for manually labeled images to obtain the features to be labeled and adjusted. Calculate the similarity between the feature to be annotated and each category center in the category center retrieval library, and take the category corresponding to the maximum similarity as the candidate automatic annotation category; write the annotation result when the maximum similarity meets the preset threshold condition, otherwise keep it in the state of waiting for manual annotation. Based on the confirmed or corrected samples, a residual feature adjuster is jointly trained using triplet loss and elastic weight solidification regularization loss, and the category center retrieval library is reconstructed according to the preset retrieval library update strategy to complete one round of incremental iterative update.

2. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, The residual feature adjuster includes at least two fully connected layers, and the residual feature adjuster is... ,in Represents the trainable parameters of the feature adjuster; The residual feature adjuster adjusts the normalized image features. Learned feature offsets: The adjusted image features can be represented as: in, Indicates the adjusted image features. This represents the feature offset learned by the residual MLP feature adjuster.

3. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, The residual feature adjuster includes at least two fully connected layers, and the weights and biases of all fully connected layers are initialized to zero; this ensures that the feature offset is always a zero vector before training begins, and the adjusted image features are exactly the same as the input normalized image features. In the initial state, the adjuster is equivalent to an identity mapping.

4. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, The category center retrieval library includes: a set of category labels, a set of sample features corresponding to each category, category center features for each category, a category center matrix, and a mapping relationship between category labels and category centers.

5. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, The preset retrieval database update strategy is a nearest neighbor center sampling strategy. For each category, the distance between the adjusted image features of each sample and the current category center is calculated. The system prioritizes retaining the samples closest to the category center as the representative feature set for that category. The category center is obtained by averaging and normalizing multiple adjusted image features within the same category. The category center is obtained by calculating the mean of image features after adjusting for similar samples and then normalizing it.

6. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, The calculation process for the candidate automatically labeled categories includes: First, the original image features are extracted and normalized using a pre-trained image encoder. Then, the normalized image features are input into a residual feature adjuster to obtain the adjusted features to be labeled. For any category in the category-centered retrieval database Calculate the features to be annotated and adjusted. With Category Center Cosine similarity between them: in, Indicates unlabeled images and categories The similarity is used to determine the category with the highest similarity, and the category corresponding to the highest similarity is used as the candidate category for automatic labeling.

7. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, In the joint training of the residual feature adjuster using the triplet loss and the elastic weight solidification regularization loss, the loss function is... ,in, The weight coefficients representing the parameter importance constraint terms. Represents the loss of the triplet. Representative parameter importance constraint loss; The loss of the triplet is: in, Indicates the interval parameter. Represents the feature distance function; , , These represent the features of the anchor sample, the positive sample belonging to the same category as the anchor sample, and the negative sample belonging to a different category from the anchor sample, respectively, after being processed by the image encoder and the residual feature adjuster. The parameter importance constraint loss is: in, These are the optimal parameters for the residual MLP feature adjuster in the previous training phase. The parameters for the current training phase are: For the first Importance coefficients of each parameter The larger the value, the higher the value. The more important a parameter is to historical labeled samples or historical category centers, the stronger the constraint on that parameter will be in subsequent incremental training.

8. The lightweight iterative human-computer collaborative annotation method based on incremental learning and dynamic updating of the retrieval database as described in claim 1, characterized in that, When reconstructing the category center retrieval database, the adjusted image features of manually confirmed and labeled images are preferentially selected as sample features in the category feature set.

9. An image incremental annotation system based on a residual feature adjuster and a category center retrieval library for implementing the method of any one of claims 1-8, characterized in that, The system includes: an image data management module, a manual annotation module, an image feature extraction module, a residual feature adjustment module, a category center retrieval library maintenance module, an automatic annotation module, an incremental update module, a result export module, and an asynchronous task management module; The image data management module is used to import and manage image data; The manual annotation module is used to receive manual annotations, confirmations, or corrections from users regarding image categories; The image feature extraction module is used to extract normalized image features through a pre-trained image encoder; The residual feature adjustment module is used to generate adjusted image features through a residual feature adjuster; The category center retrieval library maintenance module is used to maintain the category feature set and category centers based on manually confirmed samples; The automatic annotation module is used to generate automatic annotation results based on the similarity between the adjusted image features of the image to be annotated and the category center; The incremental update module is used to update the residual feature adjuster and the category center retrieval library based on manually confirmed samples.

10. An image incremental annotation device based on a residual feature adjuster and a category center retrieval library, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an image incremental annotation method based on a residual feature adjuster and a category center retrieval library as described in any one of claims 1-8.

Images