An open-source data semi-automatic data labeling method and system based on a multi-modal large model

Abstract

The application discloses a kind of open source data semi-automatic data labeling method and system based on multi-modal large model, belong to data labeling technical field.The method includes: enhancement processing;Determine image-text-category data pair training set and image-text-category data pair test set;Adjust SimCSE text encoder parameter;Adjust CLIP-SimCSE model parameter;Adjust CLIP-ViT model parameter;Test in multi-modal large model, to adjust multi-modal large model parameter;Classified labeling is carried out to determine the different image-text data pairs in the labeling result of CLIP-SimCSE model and CLIP-ViT model;Different image-text data pairs are corrected.The application can guarantee the accuracy of professional field data category labeling result, improve data labeling efficiency, significantly reduce the cost of artificial data verification.

Description

Technical Field

[0001] This invention belongs to the field of data annotation technology, and in particular relates to an open-source semi-automatic data annotation method and system based on a multimodal large model. Background Technology

[0002] With the rapid development of the internet and multimedia, open-source data on the network is becoming increasingly abundant; however, its quality varies greatly, making it difficult to use directly. By annotating open-source data, it can be transformed into high-quality, structured, and computer-readable datasets. This approach not only improves the usability and value density of the data but also provides strong support for the training and optimization of subsequent intelligent analysis systems.

[0003] Currently, open-source data on the internet is mainly presented in multimodal formats, either text or a combination of text and images. Annotation of multimodal data primarily involves semi-automatic annotation and Contrastive Language-Image Pre-training (CLIP) multimodal large-scale models. Semi-automatic data annotation is suitable for annotating open-source data in specialized fields, but it still requires significant human intervention, especially in the data annotation result verification stage, where manual calibration and verification of machine annotations remain crucial. This results in high labor costs for semi-automatic annotation.

[0004] While the CLIP model is a general-purpose, large-scale model, it is difficult to directly apply it to the annotation of open-source data in specialized fields. The CLIP model is weak in understanding fine-grained classifications or complex language. In specialized fields, this manifests as a weak ability to understand and semantically associate technical terms. Furthermore, specialized images often contain complex background information, and CLIP's general-purpose image encoder may fail to effectively extract image features, potentially leading to image-text matching errors. Additionally, when annotating image-text pairs, to ensure a strong semantic correspondence between the image-text pairs and category labels, multiple similarity calculations between the image and category labels, and between the text and category labels, are required. While this process improves the semantic consistency of the annotations, it significantly increases the computational cost of the model. Summary of the Invention

[0005] One of the objectives of this invention is to provide a semi-automated data annotation method for open-source data based on a multimodal large model. This method can ensure the accuracy of data category annotation results in professional fields, improve data annotation efficiency, and significantly reduce the cost of manual data verification.

[0006] The second objective of this invention is to provide an open-source semi-automated data annotation system based on a multimodal large model.

[0007] To achieve one of the above objectives, the present invention employs the following technical solution:

[0008] A semi-automated data annotation method for open-source data based on a multimodal large model, comprising the following steps:

[0009] Step S1: Perform enhancement processing on the original image dataset and the original text dataset in the open source data to obtain image-text data pairs;

[0010] Step S2: Label the original image-text data pairs in the image-text data pair set with categories to determine the image-text-category data pair training set and the image-text-category data pair test set;

[0011] Step S3: Input the image-text-category data pair training set into the SimCSE text encoder for training to adjust the SimCSE text encoder parameters.

[0012] Step S4: Input the image-text-category data pair training set into the CLIP-SimCSE model, which includes the parameter-adjusted SimCSE text encoder and CLIP image encoder, and perform first image feature-similar text feature alignment, first image feature-image classification head alignment, and similar text feature-text classification head alignment respectively to adjust the CLIP-SimCSE model parameters.

[0013] Step S5: Input the image-text-category data pair training set into the CLIP-ViT model, which includes the ViT image encoder and the CLIP text encoder, and perform second image feature-text feature alignment, second image feature-image classification head alignment, and text feature-text classification head alignment respectively to adjust the CLIP-ViT model parameters.

[0014] Step S6: Input the image-text-category data pair test set into the multimodal large model including the CLIP-SimCSE model and the CLIP-ViT model after parameter adjustment, and conduct tests to adjust the parameters of the multimodal large model;

[0015] Step S7: Input the unlabeled image and text data from the image-text data pair set into the multimodal large model after parameter adjustment, and perform category labeling to determine the different image-text data pairs in the labeling results of the CLIP-SimCSE model and the CLIP-ViT model;

[0016] Step S8: Correct the different image-text data pairs in the annotation results.

[0017] Furthermore, the specific implementation process of step S1 includes:

[0018] Step S11: Rotate, crop, horizontally flip, and vertically flip each image in the original image dataset from the open-source data to form an enhanced image dataset;

[0019] Step S12: Perform synonym replacement, random word insertion, and sentence rearrangement on each text in the original text dataset in the open source data to form an enhanced text dataset;

[0020] Step S13: Perform image-text matching between the original image dataset and the original text dataset and the enhanced text dataset respectively, and perform image-text matching between the enhanced image data and the original text dataset and the enhanced text dataset respectively, to form an image-text data pair set.

[0021] Furthermore, in step S3, the adjustment process of the SimCSE text encoder parameters includes:

[0022] Step S31: Identify the positive sample sentences that are semantically similar to each sentence in the text of the image-text-category data training set and all the negative sample sentences that are semantically dissimilar.

[0023] Step S32: Calculate the cosine similarity of each sentence with its corresponding positive sample sentence and all negative sample sentences to determine the loss value of each sentence;

[0024] Step S33: Calculate the mean loss of the text using the loss value of each sentence;

[0025] Step S34: Determine whether the mean loss of the text is less than the first threshold. If yes, end the training; otherwise, adjust the SimCSE text encoder parameters and return to step S31.

[0026] Furthermore, in step S4, the adjustment process of the CLIP-SimCSE model parameters includes:

[0027] Step S41: Using the CLIP image encoder, extract image features from the image-text-category data training set to form the first image feature matrix;

[0028] Step S42: Using the SimCSE text encoder, extract similar text features from the image-text-category data training set to form a similar text feature matrix;

[0029] Step S43: Align the first image feature matrix and the similar text feature matrix with image features and similar text features to obtain the first loss value when aligning the image features and similar text features.

[0030] Step S44: Align the first image feature matrix with the image classification head to obtain the second loss value when the first image feature matrix is ​​aligned with the image classification head; align the similar text feature matrix with the text classification head to obtain the third loss value when the similar text feature matrix is ​​aligned with the text classification head.

[0031] Step S45: Calculate the first total loss value of the CLIP-SimCSE model based on the first loss value, the second loss value, and the third loss value; and determine whether the first total loss value is less than the second threshold. If yes, end the training; otherwise, adjust the CLIP-SimCSE model parameters and return to step S41.

[0032] Furthermore, in step S5, the adjustment process of the CLIP-ViT model parameters includes:

[0033] Step S51: Using the ViT image encoder, extract image features from each original image in the training set using image-text-category data to form a second image feature matrix that includes global image features and local image features of each corresponding important image patch.

[0034] Step S52: Using the CLIP text encoder, extract text features from the image-text-category data training set to form a text feature matrix;

[0035] Step S53: Perform global feature alignment and local feature alignment on the second image feature matrix and the text feature matrix respectively to obtain the fourth loss value for global feature alignment and the fifth loss value for local feature alignment;

[0036] Step S54: Calculate the sixth loss value of the ViT image encoder based on the fourth and fifth loss values; and determine whether the sixth loss value is less than the third threshold. If yes, end the training and use the sixth loss value of the ViT image encoder as the sixth loss value obtained after training, and proceed to step S55; otherwise, adjust the parameters of the ViT image encoder and return to step S51.

[0037] Step S55: Align the second image feature matrix with the image classification head to obtain the seventh loss value when the second image feature is aligned with the image classification head; align the text feature matrix with the text classification head respectively to obtain the eighth loss value when the text feature is aligned with the text classification head.

[0038] Step S56: Calculate the second total loss value of the CLIP-ViT model based on the sixth, seventh, and eighth loss values ​​obtained after training; and determine whether the second total loss value is less than the fourth threshold. If yes, end the training; otherwise, adjust the CLIP-ViT model parameters and return to step S55.

[0039] Furthermore, the specific implementation process of step S51 includes:

[0040] Step S511: For each original image in the training set, the image-text-category data is sequentially divided into blocks and vectorized to form an image block vector sequence;

[0041] Step S512: Add an image category to the beginning of the image block vector sequence to form a new vector sequence;

[0042] Step S513: Add a positional code to the image category and each image block in the new vector sequence to form the image input sequence;

[0043] Step S514: Input the image input sequence into the Transformer encoder to obtain global image features and local image features of each image patch;

[0044] Step S515: Using global image features and local image features of each image patch, an attention mechanism is employed to calculate the contribution of each image patch to the global image features, in order to identify important image patches whose contribution is greater than the fifth threshold.

[0045] Step S516: Use global image features and local image features of each important image block to form a second image feature matrix.

[0046] Furthermore, in step S6, the testing process for the multimodal large model includes:

[0047] Step S61: Using the CLIP-SimCSE model, after extracting image features and similar text features from the image-text-category data test set respectively, perform image feature-image classification head alignment and similar text feature-text classification head alignment respectively to determine the first test accuracy of the CLIP-SimCSE model.

[0048] Step S62: Using the CLIP-ViT model, image features and text features are extracted from the image-text-category data of the test set, and then image feature-image classification head alignment and text feature-text classification head alignment are performed to determine the second test accuracy of the CLIP-ViT model.

[0049] Step S63: Determine whether the accuracy of the first test and the accuracy of the second test are both greater than or equal to the sixth threshold. If yes, end the test; otherwise, adjust the model parameters for the model with an accuracy less than the sixth threshold and return to the corresponding model training process.

[0050] Furthermore, the open-source semi-automated data annotation method also includes:

[0051] Step S9: Add the corrected different image-text data pairs to the image-text-category data pair training set to update the image-text-category data pair training set.

[0052] To achieve the second objective mentioned above, the present invention employs the following technical solution:

[0053] An open-source semi-automated data annotation system based on a multimodal large model, the open-source semi-automated data annotation system comprising:

[0054] The enhancement processing module is used to perform enhancement processing on the original image dataset and the original text dataset in the open source data respectively to obtain image-text data pairs;

[0055] The first category labeling module is used to label the original image-text data pairs in the image-text data pair set with categories in order to determine the image-text-category data pair training set and the image-text-category data pair test set;

[0056] The first training module is used to input the image-text-category data pair training set into the SimCSE text encoder for training, so as to adjust the parameters of the SimCSE text encoder.

[0057] The first alignment module is used to input the image-text-category data pair training set into the CLIP-SimCSE model, which includes the parameter-adjusted SimCSE text encoder and CLIP image encoder, and to perform first image feature-similar text feature alignment, first image feature-image classification head alignment, and similar text feature-text classification head alignment, respectively, so as to adjust the CLIP-SimCSE model parameters.

[0058] The second alignment module is used to input the image-text-category data pair training set into the CLIP-ViT model, which includes the ViT image encoder and the CLIP text encoder, and to perform second image feature-text feature alignment, second image feature-image classification head alignment, and text feature-text classification head alignment, respectively, so as to adjust the CLIP-ViT model parameters.

[0059] The testing module is used to input the image-text-category data pair test set into the multimodal large model, which includes the CLIP-SimCSE model and the CLIP-ViT model with parameter adjustments, respectively, for testing, so as to adjust the parameters of the multimodal large model;

[0060] The second category labeling module is used to input unlabeled image and text data from the image-text data pair set into the parameter-adjusted multimodal large model for category labeling, so as to identify different image-text data pairs in the labeling results of the CLIP-SimCSE model and the CLIP-ViT model;

[0061] The correction module is used to correct different image-text data pairs in the annotation results.

[0062] Furthermore, the open-source semi-automated data annotation system also includes:

[0063] The update module is used to add the corrected and different image-text data pairs to the image-text-class data pair training set to update the image-text-class data pair training set.

[0064] In summary, the solution proposed in this invention has the following technical effects:

[0065] This invention improves annotation performance by using the SimCSE text encoder in the CLIP-SimCSE model to shorten the distance between semantically similar samples while widening the distance between semantically dissimilar samples. It also improves similarity by using the ViT image encoder in the CLIP-ViT model to force the CLIP-ViT model to focus on local regions (i.e., local image features), avoiding the problem of the CLIP image encoder's global attention mechanism ignoring local image details. Furthermore, it achieves strong semantic correspondence between image-text data pairs and category labels through image-image classification head alignment and text-text classification head alignment, reducing large-scale similarity calculations and the computational burden of multimodal large models. Finally, it utilizes the CLIP-SimCSE model, which exhibits both similarity and difference, and the C... The LIP-ViT model reduces the computational cost of two large models and avoids the CLIP-SimCSE and CLIP-ViT models producing the same incorrect annotation results. Both CLIP-SimCSE and CLIP-ViT models are based on CLIP, and the text and image features extracted by this model are located in a shared semantic space, enabling direct semantic comparison and interaction between the two types of features, providing a foundation for solving cross-modal alignment problems. This invention achieves mutual calibration of data annotation results through the CLIP-SimCSE and CLIP-ViT models, improving the efficiency and accuracy of data annotation results. While ensuring the efficiency and accuracy of data annotation results, this invention reduces human intervention as the large model is continuously optimized, thus lowering labor costs. Attached Figure Description

[0066] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 This is a flowchart illustrating the open-source semi-automated data annotation method based on a multimodal large model according to the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] This embodiment presents a semi-automated data annotation method for open-source data based on a multimodal large model, referencing... Figure 1 This open-source, semi-automated data annotation method includes the following steps:

[0070] Step S1: Perform enhancement processing on the original image dataset and the original text dataset in the open source data to obtain image-text data pairs.

[0071] Data augmentation is performed on the pre-processed original image-text data pairs. The augmented images and text are then matched with the original images and text to form original image-text data pairs, augmented image-text data pairs, and other data pairs. Data augmentation includes image data augmentation and text data augmentation. Image data augmentation methods include rotation, cropping, and flipping, while text data augmentation methods include synonym replacement, random insertion, and sentence rearrangement.

[0072] Image data augmentation, assuming the original image dataset N represents the number of original images, the nth image in the original image dataset. Zhang Original Image It is A two-dimensional matrix, representing the first... The pixel values ​​of the original image are represented as follows:

[0073] (1)

[0074] in, For the first image in the original image dataset Zhang's original image; For the first image in the original image dataset Zhang's original image at location Pixel value at; and The first The height and width of the original image.

[0075] Rotate the image around its center point by an angle. Generate a new image. Let the original image be... The center point is , , Rotation angle They are usually chosen randomly, with an angle range of [missing information]. The rotation operation will change each position. Map to new location :

[0076] (2)

[0077] Rotated image The rotated pixel values:

[0078] ; (3)

[0079] in, These are the position coordinates of the image after rotation; This represents the pixel value at the position corresponding to the rotated image.

[0080] A sub-region is selected from the image to generate the enhanced image. Assume the size of the cropped region is... The cropping operation selects a cropping frame. Typically, the image is cropped from the center, and the cropping box has the following boundaries:

[0081] (4)

[0082] in, For the cropping frame The center coordinates; and Crop boxes Width and height, cropped image for:

[0083] ; (5)

[0084] in, For the cropping frame The pixel coordinates within the cropping box. The specific position of the cropping box can be randomly selected or fixed.

[0085] Flip the image (either horizontally or vertically) to generate an enhanced image. Assume a horizontal or vertical flip is chosen. This is achieved by adjusting each coordinate of the image... Mapped to Achieve horizontal flipping by adjusting each coordinate of the image. Mapped to Achieve vertical flipping. (Image after horizontal flipping) It can be represented as:

[0086] ; (6)

[0087] Image after vertical flip It can be represented as:

[0088] ; (7)

[0089] The choice of flip operation is usually random; it can be a horizontal flip, a vertical flip, or both.

[0090] The resulting augmented image dataset after rotation, cropping, and flipping operations. for:

[0091] ; (8)

[0092] in, To obtain the original image Enhanced image obtained after rotation, cropping, and flipping operations.

[0093] Text data augmentation, assuming the input text dataset Each text It is by A sequence of words is denoted as:

[0094] ; (9)

[0095] in, For the first The first text in the text one word, and , It is a vocabulary list.

[0096] By certain words in the text Replace the words with their synonyms (i.e., synonym replacement) to generate augmented data. Define a set of synonyms ,Right now:

[0097] ; (10)

[0098] in, yes Synonyms, define a probability This indicates whether to perform a word check. The probability of performing a synonym replacement. If replacement is selected, then from... Randomly select a synonym replace Synonym substitution can be represented as:

[0099] ; (11)

[0100] in, The probability of a replacement operation occurring is determined by Decide.

[0101] Enhance the text by randomly selecting a position and inserting a random word (i.e., random insertion). Suppose that in the text... Randomly select a position (i.e., random selection) And select a word. Insert. The insert operation will insert text. Transform into:

[0102] ;(12)

[0103] in, From the vocabulary list A word is randomly selected from the given text. The probability of the insertion operation. Indicates whether to perform the insertion operation.

[0104] Augmented data is generated by changing the order of words in the text (i.e., sentence rearrangement). Let the text... Include one word Sentence rearrangement generates new text by randomly arranging the words in the original text. There is a random permutation function. Its function is to rearrange the sequence of words to obtain a new order:

[0105] ; (13)

[0106] Then, the text The words are rearranged in a new order to obtain the enhanced text:

[0107] ;(14)

[0108] in, It is a random permutation function.

[0109] An enhanced text dataset is generated by performing synonym replacement, random insertion, or sentence rearrangement on each text data point. ,Right now:

[0110] ; (15)

[0111] in, Enhanced text data is obtained by applying any of the above enhancement operations (synonym replacement, random insertion, or sentence rearrangement).

[0112] After enhancing the original image dataset and the original text dataset, the original image dataset, the original text dataset, the enhanced image dataset, and the enhanced text dataset are then matched with "image-text" data pairs.

[0113] The original image dataset is known to be The corresponding original text dataset Then the original image-original text data pair can be represented as:

[0114] ; (16)

[0115] The enhanced image dataset is Enhance the text dataset for Then, the enhanced image-original text data pair, the original image-enhanced text data pair, and the enhanced image-enhanced text data pair are respectively:

[0116] ; (17)

[0117] ; (18)

[0118] ; (19)

[0119] In summary, the specific implementation process of this step includes:

[0120] Step S11: Rotate, crop, horizontally flip, and vertically flip each image in the original image dataset from the open-source data to form an enhanced image dataset;

[0121] Step S12: Perform synonym replacement, random word insertion, and sentence rearrangement on each text in the original text dataset in the open source data to form an enhanced text dataset;

[0122] Step S13: Perform image-text matching between the original image dataset and the original text dataset and the enhanced text dataset respectively, and perform image-text matching between the enhanced image data and the original text dataset and the enhanced text dataset respectively, to form an image-text data pair set.

[0123] Step S2: Label the original image-text data pairs in the image-text data pair set with categories to determine the image-text-category data pair training set and the image-text-category data pair test set.

[0124] For the original image-original text data pair Manual annotation ensures the accuracy of the original real data. The annotated data is used to train and test the large model, reducing the interference of data noise on the performance of the multimodal large model.

[0125] Set the data annotation format as follows: ,in, For the first The original image data in each original image-original text data pair For the first The original text data in each original image-original text data pair For the first The original image-text data pairs contain the categories to which the images and text belong. The labeled data is then divided into training sets. and test set Among them, the training set Includes Data in batches, each batch containing One original image-original text data pair { }

[0126] Step S3: Input the image-text-category data pair training set into the SimCSE text encoder for training to adjust the SimCSE text encoder parameters.

[0127] Before the SimCSE text encoder is used in the CLIP model, it needs to be trained to achieve superior performance in converting text into vectors. This superior performance during training manifests as ensuring that semantically similar sentences are closer together in the vector space. The SimCSE text encoder uses contrastive learning to bring similar samples closer together while widening the distance between dissimilar samples.

[0128] In the Each text in each batch All contain There are several sentences, and the relationships between them are unknown. Generally, the relationships between sentences are described using implied, contradictory, or neutral terms. When using SimCSE for unsupervised training of sentences, a positive sample is first constructed for each sentence—a sample that has the same semantics as the original sentence but with subtle differences in its vector representation. The sentences are then set up... The positive sample sentence is Then the relationship between the two sentences is implied. All other sentences, and the positive samples constructed for other sentences, are sentences. and Negative samples, they are related to sentences and The relationship is either contradictory or neutral. In each sentence, each positive sample pair All have One negative sample.

[0129] The SimCSE text encoder uses InfoCSE Loss as the loss function for contrastive learning to measure the model's training performance. When the loss value (i.e., the text...) is... The mean loss for each text middle The SimCSE text encoder training is complete when the mean loss value of each sentence is minimized or is less than the set threshold (i.e., the first threshold). The loss function is shown below:

[0130] ; (20)

[0131] in, For the first The loss value of each sentence; is the cosine similarity function, used to calculate the similarity score between two sentences. The denominator in formula (20) calculates the positive samples. The sum of similarities with all sentences in the batch (including itself and all negative samples). This forces the model to both bring positive sample pairs closer together and push away all negative samples. Finally, minimize The average loss of each sentence completes the entire training process.

[0132] In summary, the adjustment process of the SimCSE text encoder parameters in this embodiment includes:

[0133] Step S31: Identify the positive sample sentences that are semantically similar to each sentence in the text of the image-text-category data training set and all the negative sample sentences that are semantically dissimilar.

[0134] Step S32: Calculate the cosine similarity of each sentence with its corresponding positive sample sentence and all negative sample sentences to determine the loss value of each sentence;

[0135] Step S33: Calculate the mean loss of the text using the loss value of each sentence;

[0136] Step S34: Determine whether the mean loss of the text is less than the first threshold. If yes, end the training; otherwise, adjust the SimCSE text encoder parameters and return to step S31.

[0137] Step S4: Input the image-text-category data pair training set into the CLIP-SimCSE model, which includes the parameter-adjusted SimCSE text encoder and CLIP image encoder, and perform first image feature-similar text feature alignment, first image feature-image classification head alignment, and similar text feature-text classification head alignment respectively to adjust the CLIP-SimCSE model parameters.

[0138] Image-text matching is performed by calculating the similarity between image features and similar text features. The goal is to maximize the similarity between related images and text or minimize the contrast loss between them, thus completing model training. This embodiment uses cosine similarity to measure the similarity between image features and text features. The image feature matrix and text feature matrix are as follows:

[0139] ; (twenty one)

[0140] ; (twenty two)

[0141] in, and These are the image feature matrix and the text feature matrix, respectively. For the first The first image Image features; For the first The first text Image features; , is the dimension of the text feature vector.

[0142] This embodiment achieves image-text data matching by using the similarity between images and text:

[0143] ;(twenty three)

[0144] in, For the first The image and the first The similarity of individual texts.

[0145] The loss value for image feature-text feature alignment in this embodiment is:

[0146] ; (twenty four)

[0147] ; (25)

[0148] ; (26)

[0149] in, The loss value for image feature-text feature alignment; and These are the image-to-text loss value and the text-to-image loss value, respectively. For the first The image and the first The diagonal element values ​​in the similarity matrix of the texts.

[0150] The image-text similarity matrix in this embodiment for:

[0151] ;(27)

[0152] By minimizing This allows for the alignment of paired images and text in the feature space, thus completing the training of the model.

[0153] The trained SimCSE text encoder and CLIP image encoder are combined to process image-text data pairs, mapping similar text features and image features to the same shared semantic space. Then, their cosine similarity is calculated to determine their degree of association.

[0154] category It is a word or phrase that cannot be directly input into the text encoder. This embodiment will... Convert this into a sentence, such as "This is a photo of XX, and a text description of it" (or other sentence structures are also possible), where "XX" is... To reduce the amount of computation, indivual Perform duplicate value removal, and then... The sentences are transformed into unique categories, and the transformed sentences are then input into a text encoder.

[0155] To facilitate similarity calculation, the category matrix is... Set as Dimension, category matrix for:

[0156] ; (28)

[0157] To achieve a strong semantic correspondence between image-text data pairs and categories, it is necessary to calculate the similarity between images and text, images and categories, and text and categories separately, and then calculate a loss function based on the similarity. Minimizing the loss function allows for the optimization of the large model, completing the model training. However, calculating three similarity matrices means storing three times the number of intermediate variables as the original large model and performing three backpropagation steps. This significantly increases memory consumption and computation time.

[0158] To reduce computational load, this embodiment uses features output by the image encoder. Then, a deep neural network model is added as the image classification head. The final layer of the model uses the softmax function as the activation function to obtain multi-class probability values. Similarly, the features output by the text encoder... Subsequently, a deep neural network model was added as the text classification head. These two deep neural network models differ in their parameters, but the output dimension of both classification heads is the total number of classes. (Number of unique categories in this batch of data). Deep neural network models are used as classification heads (i.e., image classification head and text classification head). On the one hand, compared to traditional machine learning models, deep neural network models can perform multi-class recognition; on the other hand, compared to large language models, deep neural network models have lower computational cost and higher computational efficiency. The cross-entropy loss functions for these two classification heads are as follows:

[0159] ; (29)

[0160] ; (30)

[0161] in, and These are the loss values ​​for image feature-image classification head alignment and text feature-text classification head alignment, respectively. and These are the image category probability and the text category probability, respectively; For true category labels, use one-hot encoding and and The loss is minimized to complete the training of the classification head.

[0162] Based on the above, the adjustment process of CLIP-SimCSE model parameters includes:

[0163] Step S41: Using the CLIP image encoder, extract image features from the image-text-category data training set to form the first image feature matrix.

[0164] Step S42: Using the SimCSE text encoder, extract similar text features from the image-text-category data training set to form a similar text feature matrix;

[0165] Step S43: Align the first image feature matrix and the similar text feature matrix with image features and similar text features to obtain the first loss value when aligning the image features and similar text features.

[0166] Step S44: Align the first image feature matrix with the image classification head to obtain the second loss value when the first image feature matrix is ​​aligned with the image classification head; align the similar text feature matrix with the text classification head to obtain the third loss value when the similar text feature matrix is ​​aligned with the text classification head.

[0167] Step S45: Calculate the first total loss value of the CLIP-SimCSE model based on the first loss value, the second loss value, and the third loss value; and determine whether the first total loss value is less than the second threshold. If yes, end the training; otherwise, adjust the CLIP-SimCSE model parameters and return to step S41.

[0168] The first total loss value of the CLIP-SimCSE model in this embodiment is:

[0169] ; (31)

[0170] in, This is the first total loss value for the CLIP-SimCSE model; The first loss value when aligning image features with similar text features; The second loss value is used when aligning the first image feature with the image classification head. The third loss value is used when aligning similar text features with the text classification head. These are the parameters for the CLIP-SimCSE model.

[0171] This is the CLIP-SimCSE model, used to balance the weights of contrastive and classification losses, with values ​​ranging from [0.1-1.0]. The computational cost of cross-entropy loss is much less than that of calculating the contrastive loss between two similarity matrices. Compared to the original CLIP model, it only adds two linear layers of forward propagation, yet achieves a strong semantic correspondence between image-text data pairs and category labels. Finally, by minimizing... Complete the training of the CLIP-SimCSE model.

[0172] Step S5: Input the image-text-category data pair training set into the CLIP-ViT model, which includes the ViT image encoder and the CLIP text encoder, and perform second image feature-text feature alignment, second image feature-image classification head alignment, and text feature-text classification head alignment respectively to adjust the CLIP-ViT model parameters.

[0173] Since the CLIP global attention mechanism typically ignores local image details, it leads to biases in similarity calculation. Therefore, the CLIP-ViT model in this embodiment uses Enhanced Local Awareness ViT as the image encoder, adding an auxiliary loss to the ViT patch embedding to force the model to focus on local regions. The CLIP-ViT model parameter adjustment process includes:

[0174] Step S51: Using the ViT image encoder, extract image features from each original image in the training set of the image-text-category data to form a second image feature matrix that includes global image features and local image features of each corresponding important image patch.

[0175] In this embodiment, the specific implementation process of this step includes:

[0176] Step S511: For the image-text-category data, each original image in the training set is sequentially divided into blocks and vectorized to form an image block vector sequence.

[0177] Assuming the original image The size is ,in, For height, For width, The number of color channels. The original image is segmented into... Individual Image patches Each image patch is flattened and mapped to a fixed-dimensional vector through a linear layer. Therefore, the initial patch vector sequence is... .

[0178] Step S512: Add an image category at the beginning of the image block vector sequence to form a new vector sequence.

[0179] Add category tokens and location encoding. The CLIP-ViT model is trained using labeled image-text pairs, with each image having its own category label. .

[0180] Step S513: Add a positional code to the image category and each image block in the new vector sequence to form the image input sequence.

[0181] Add a positional code to the vector of each image patch. To mark the position of each image patch, the image... The input sequence is:

[0182] ; (32)

[0183] Step S514: Input the image input sequence into the Transformer encoder to obtain global image features and local image features of each image block.

[0184] Subsequently, the Transformer encoder processes and extracts features. Let the Transformer encoder have... A multi-layer neural network, where each layer contains a multi-head self-attention mechanism and a feedforward neural network, along with residual connections and layer normalization. The image input sequence... The output sequence obtained after inputting into the Transformer encoder is shown below:

[0185] ; (33)

[0186] in, The output sequence; Global image features, representing category labels The final state is the global feature representation of the image. For the first Image blocks The corresponding local image features include local detail information.

[0187] Step S515: Using global image features and local image features of each image block, an attention mechanism is employed to calculate the contribution of each image block to the global image features, in order to identify important image blocks whose contribution is greater than the fifth threshold.

[0188] Unlike directly using the ViT image encoder for image feature extraction, this embodiment adds an auxiliary loss to the patch embedding of the ViT image encoder, forcing the model to focus on local regions. Therefore, the loss function must consider not only the global loss but also the local loss.

[0189] Considering the Calculating loss values ​​for each image patch would significantly increase the computational cost of the model. This embodiment employs attention-based importance sampling. (Calculate category labels.) Attention weights for all patches These weights reflect the contribution of each image patch to the final global decision. The patch with the highest attention weight is selected. These are key image patches. These are regions that the model considers "important," and they are likely to have a higher relevance to the text description. The greatest benefit is achieved by enhancing the alignment of key image patches. Only for... Calculating auxiliary loss for each important image patch can significantly reduce the computational load of the model.

[0190] Step S516: Use global image features and local image features of each important image block to form a second image feature matrix.

[0191] Step S52: Using the CLIP text encoder, extract text features from the image-text-category data training set to form a text feature matrix.

[0192] Step S53: Perform global feature alignment and local feature alignment on the second image feature matrix and the text feature matrix respectively to obtain the fourth loss value for global feature alignment and the fifth loss value for local feature alignment.

[0193] Fourth loss value for global feature alignment Through calculation The loss is obtained by comparing the similarity matrix between the projected features and the text features. This is the fifth loss value for local feature alignment. ,calculate The similarity between the features of important image patches after projection and the features of the same text is calculated. For each important image patch, it is treated as an independent "image representation", and a contrastive loss is calculated with all texts in the batch, and then averaged.

[0194] Step S54: Calculate the sixth loss value of the ViT image encoder based on the fourth and fifth loss values; and determine whether the sixth loss value is less than the third threshold. If yes, end the training and use the sixth loss value of the ViT image encoder as the sixth loss value obtained after training, and proceed to step S55; otherwise, adjust the parameters of the ViT image encoder and return to step S51.

[0195] The sixth loss value of the ViT image encoder in this embodiment (That is, the total loss value of the ViT image encoder) is shown below:

[0196] ; (34)

[0197] in, This is the sixth loss value of the ViT image encoder; and These are the fourth loss value for global feature alignment and the fifth loss value for local feature alignment, respectively. For the ViT image encoder parameters, a small value should be taken to avoid local loss dominating training.

[0198] We combine a fully trained enhanced local perception ViT image encoder with a CLIP text encoder to process image-text data pairs, mapping image features and text features to the same shared semantic space, and then calculating their cosine similarity to determine their degree of association.

[0199] Step S55: Align the second image feature matrix with the image classification head to obtain the seventh loss value when the second image feature matrix is ​​aligned with the image classification head; align the text feature matrix with the text classification head respectively to obtain the eighth loss value when the text feature matrix is ​​aligned with the text classification head.

[0200] Step S56: Calculate the second total loss value of the CLIP-ViT model based on the sixth, seventh, and eighth loss values ​​obtained after training; and determine whether the second total loss value is less than the fourth threshold. If yes, end the training; otherwise, adjust the CLIP-ViT model parameters and return to step S55.

[0201] The second total loss value of the CLIP-ViT model in this embodiment is:

[0202] ; (35)

[0203] in, This is the second total loss value for the CLIP-ViT model; This is the sixth loss value of the ViT image encoder at the end of training (i.e., the loss value when aligning the image and text features of the second image feature matrix and text feature matrix). and These are the seventh loss value when aligning the second image feature with the image classification head and the eighth loss value when aligning the text feature with the text classification head, respectively. The parameters for the CLIP-ViT model are used to balance the weights of the contrastive and classification losses, with values ​​ranging from [0.1-1.0]. The computational cost of the cross-entropy loss is much less than that of the contrastive loss, which calculates two similarity matrices. Compared to the original CLIP model, it only adds two linear layers of forward propagation, yet achieves a strong semantic correspondence between image-text data pairs and category labels. Finally, by minimizing... Complete the training of the CLIP-ViT model.

[0204] Step S6: Input the image-text-category data pair test set into the multimodal large model, which includes the CLIP-SimCSE model and the CLIP-ViT model with adjusted parameters, respectively, and conduct tests to adjust the parameters of the multimodal large model.

[0205] The image-text data pairs in the test set are tested using image-text-category data. First, the CLIP-SimCSE and CLIP-ViT models are used to calculate feature vectors for images and text, respectively. Then, these feature vectors are input into the classification head to obtain the image category probability. and text category probability Take the average of the results The category to which the maximum value belongs is the final result.

[0206] ; (36)

[0207] By comparing the true and labeled categories of all data pairs in the test set, the accuracy of the CLIP-SimCSE and CLIP-ViT models on the image-text-category data pair test set (i.e., the first test accuracy and the second test accuracy) is obtained. If the accuracy of both large models (i.e., the CLIP-SimCSE model and the CLIP-ViT model) is greater than the set value... If the model does not pass the test, the test ends; otherwise, the model parameters are adjusted and the test is repeated.

[0208] In summary, the testing process for the multimodal large model in this embodiment includes:

[0209] Step S61: Using the CLIP-SimCSE model, after extracting image features and similar text features from the image-text-category data test set respectively, perform image feature-image classification head alignment and similar text feature-text classification head alignment respectively to determine the first test accuracy of the CLIP-SimCSE model.

[0210] Step S62: Using the CLIP-ViT model, image features and text features are extracted from the image-text-category data of the test set, and then image feature-image classification head alignment and text feature-text classification head alignment are performed to determine the second test accuracy of the CLIP-ViT model.

[0211] Step S63: Determine whether the accuracy of the first test and the accuracy of the second test are both greater than or equal to the sixth threshold. If yes, end the test; otherwise, adjust the model parameters for the model with an accuracy less than the sixth threshold and return to the corresponding model training process.

[0212] Step S7: Input the unlabeled image and text data from the image-text data pair set into the multimodal large model after parameter adjustment, and perform category labeling to determine the different image-text data pairs in the labeling results of the CLIP-SimCSE model and the CLIP-ViT model.

[0213] The unlabeled data were labeled using the tested multimodal large models (i.e., CLIP-SimCSE model and CLIP-ViT model), and two labeling results were obtained for each data pair. and By comparing the two annotation results, the final data annotation is obtained.

[0214] Set the unlabeled image dataset as Unlabeled text dataset . Pair the data Input CLIP-SimCSE and CLIP-ViT models respectively to obtain data annotation results. and The final data annotation results are obtained by comparing the data using the following formula.

[0215] ; (37)

[0216] If the two annotation results are the same If the results are different, save the results and complete the data annotation; if the two results are different... The CLIP-SimCSE and CLIP-ViT models contain different image-text data pairs, requiring manual annotation. Add annotations.

[0217] Step S8: Correct the different image-text data pairs in the annotation results.

[0218] For different image-text data pairs in the corrected (e.g., manually labeled) annotation results, the data is retrained in both multimodal large models after correct annotation is completed to optimize the models until the two large models have completely consistent annotation results for the data pairs. Therefore, the open-source semi-automated data annotation method in this embodiment also includes:

[0219] Step S9: Add the corrected different image-text data pairs to the image-text-category data pair training set to update the image-text-category data pair training set.

[0220] This embodiment improves annotation performance by using the SimCSE text encoder in the CLIP-SimCSE model to shorten the distance between semantically similar samples and widen the distance between semantically dissimilar samples. It also improves similarity by using the ViT image encoder in the CLIP-ViT model to force the CLIP-ViT model to focus on local regions (i.e., local image features), avoiding the problem of the CLIP image encoder's global attention mechanism ignoring local image details. Furthermore, this embodiment achieves a strong semantic correspondence between image-text data pairs and category labels through image-image classification head alignment and text-text classification head alignment, reducing large-scale similarity calculations and the computational burden of multimodal large models. Finally, this embodiment utilizes the CLIP-SimCSE model, which exhibits both similarity and difference, and... The CLIP-ViT model reduces the computational cost of two large models and avoids the CLIP-SimCSE and CLIP-ViT models producing the same incorrect annotation results. Both CLIP-SimCSE and CLIP-ViT models are based on CLIP, and the text and image features extracted by this model are located in a shared semantic space, enabling direct semantic comparison and interaction between the two types of features, providing a foundation for solving cross-modal alignment problems. This embodiment achieves mutual calibration of data annotation results through the CLIP-SimCSE and CLIP-ViT models, improving the efficiency and accuracy of data annotation results. Under the condition of ensuring the efficiency and accuracy of data annotation results, as the large model is continuously optimized, the need for human intervention will decrease, reducing labor costs.

[0221] The technical solutions of the above embodiments can be implemented using the technical solutions given in the following embodiments:

[0222] An open-source semi-automated data annotation system based on a multimodal large model, comprising:

[0223] The enhancement processing module is used to perform enhancement processing on the original image dataset and the original text dataset in the open source data respectively to obtain image-text data pairs;

[0224] The first category labeling module is used to label the original image-text data pairs in the image-text data pair set with categories in order to determine the image-text-category data pair training set and the image-text-category data pair test set;

[0225] The first training module is used to input the image-text-category data pair training set into the SimCSE text encoder for training, so as to adjust the parameters of the SimCSE text encoder.

[0226] The first alignment module is used to input the image-text-category data pair training set into the CLIP-SimCSE model, which includes the parameter-adjusted SimCSE text encoder and CLIP image encoder, and to perform first image feature-similar text feature alignment, first image feature-image classification head alignment, and similar text feature-text classification head alignment, respectively, so as to adjust the CLIP-SimCSE model parameters.

[0227] The second alignment module is used to input the image-text-category data pair training set into the CLIP-ViT model, which includes the ViT image encoder and the CLIP text encoder, and to perform second image feature-text feature alignment, second image feature-image classification head alignment, and text feature-text classification head alignment, respectively, so as to adjust the CLIP-ViT model parameters.

[0228] The testing module is used to input the image-text-category data pair test set into the multimodal large model, which includes the CLIP-SimCSE model and the CLIP-ViT model with parameter adjustments, respectively, for testing, so as to adjust the parameters of the multimodal large model;

[0229] The second category labeling module is used to input unlabeled image and text data from the image-text data pair set into the parameter-adjusted multimodal large model for category labeling, so as to identify different image-text data pairs in the labeling results of the CLIP-SimCSE model and the CLIP-ViT model;

[0230] The correction module is used to correct different image-text data pairs in the annotation results.

[0231] Furthermore, the enhancement process includes:

[0232] The first forming submodule is used to rotate, crop, horizontally flip, and vertically flip each image in the original image dataset in the open source data to form an enhanced image dataset;

[0233] The second submodule is used to perform synonym replacement, random word insertion, and sentence rearrangement on each text in the original text dataset in the open source data to form an enhanced text dataset;

[0234] The image-text matching submodule is used to perform image-text matching between the original image dataset and the original text dataset and the enhanced text dataset, respectively, and to perform image-text matching between the enhanced image data and the original text dataset and the enhanced text dataset, respectively, to form image-text data pairs.

[0235] Furthermore, the first training module includes:

[0236] The determination submodule is used to determine the positive sample sentences that are semantically similar to each sentence in the text of the image-text-category data pair training set and all negative sample sentences that are semantically dissimilar.

[0237] The first calculation submodule is used to calculate the cosine similarity of each sentence with its corresponding positive sample sentences and all negative sample sentences, so as to determine the loss value of each sentence;

[0238] The second calculation submodule is used to calculate the average loss of the text using the loss value of each sentence;

[0239] The first judgment submodule is used to determine whether the mean loss of the text is less than a first threshold. If so, the training ends; if not, the SimCSE text encoder parameters are adjusted and transmitted to the determination submodule.

[0240] Furthermore, the first alignment module includes

[0241] CLIP image encoder is used to extract image features from the image-text-category data training set to form the first image feature matrix;

[0242] SimCSE text encoder is used to extract similar text features from image-text-category data training sets to form a similar text feature matrix;

[0243] The first alignment submodule is used to perform image feature-similar text feature alignment on the first image feature matrix and the similar text feature matrix to obtain a first loss value when image feature-similar text feature alignment is performed; and to align the first image feature matrix with the image classification head to obtain a second loss value when first image feature-image classification head alignment is performed; and to align the similar text feature matrix with the text classification head to obtain a third loss value when similar text feature-text classification head alignment is performed.

[0244] The second judgment submodule is used to calculate the first total loss value of the CLIP-SimCSE model based on the first loss value, the second loss value, and the third loss value; and to determine whether the first total loss value is less than the second threshold. If so, the training ends; if not, the CLIP-SimCSE model parameters are adjusted and transmitted to the SimCSE text encoder.

[0245] Furthermore, the second alignment module

[0246] The ViT image encoder is used to extract image features from each original image in the training set of the image-text-category data pair to form a second image feature matrix that includes global image features and local image features of each corresponding important image patch.

[0247] CLIP text encoder is used to extract text features from image-text-category data training sets to form a text feature matrix;

[0248] The second alignment submodule is used to perform global feature alignment and local feature alignment on the second image feature matrix and the text feature matrix respectively, so as to obtain the fourth loss value when global feature alignment and the fifth loss value when local feature alignment.

[0249] The third judgment submodule is used to calculate the sixth loss value of the ViT image encoder based on the fourth and fifth loss values; and to determine whether the sixth loss value is less than the third threshold. If so, the training ends and the sixth loss value of the ViT image encoder is transmitted to the third alignment submodule as the sixth loss value obtained after training. If not, the parameters of the ViT image encoder are adjusted and transmitted to the ViT image encoder.

[0250] The third alignment submodule is used to align the second image feature matrix with the image classification head to obtain the seventh loss value when the second image feature is aligned with the image classification head, and to align the text feature matrix with the text classification head respectively to obtain the eighth loss value when the text feature is aligned with the text classification head.

[0251] The fourth judgment submodule is used to calculate the second total loss value of the CLIP-ViT model based on the sixth, seventh, and eighth loss values ​​obtained after training; and to determine whether the second total loss value is less than the fourth threshold. If so, training ends; otherwise, the CLIP-ViT model parameters are adjusted and transmitted to the third alignment submodule.

[0252] Furthermore, the ViT image encoder includes:

[0253] The first processing subunit is used to sequentially divide and vectorize each original image in the training set of the image-text-category data to form an image block vector sequence.

[0254] The second processing subunit is used to add the image category at the beginning of the image block vector sequence to form a new vector sequence;

[0255] The third processing subunit is used to add a positional code to the image category and each image block in the new vector sequence to form the image input sequence;

[0256] The Transformer encoder is used to input image input sequences into the Transformer encoder to obtain global image features and local image features of each image patch;

[0257] The computational subunit is used to utilize global image features and local image features of each image patch, employing an attention mechanism, to calculate the contribution of each image patch to the global image features, in order to identify important image patches whose contribution is greater than the fifth threshold.

[0258] The third processing subunit is used to form a second image feature matrix by utilizing global image features and local image features of each important image block.

[0259] Furthermore, the testing module includes:

[0260] The CLIP-SimCSE model is used to extract image features and similar text features from the image-text-category data test set, and then perform image feature-image classification head alignment and similar text feature-text classification head alignment to determine the first test accuracy of the CLIP-SimCSE model.

[0261] The CLIP-ViT model is used to extract image features and text features from the image-text-category data test set, and then perform image feature-image classification head alignment and text feature-text classification head alignment to determine the second test accuracy of the CLIP-ViT model.

[0262] The fifth judgment submodule is used to determine whether the accuracy of the first test and the accuracy of the second test are both greater than or equal to the sixth threshold. If so, the test ends; if not, the model parameters of the model with a test accuracy less than the sixth threshold are adjusted, and the corresponding model training process is returned.

[0263] Furthermore, the open-source semi-automated data annotation system also includes:

[0264] The update module is used to add the corrected and different image-text data pairs to the image-text-class data pair training set to update the image-text-class data pair training set.

[0265] The principles, formulas, and parameter definitions involved in the above embodiments are all applicable and will not be repeated here.

[0266] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A semi-automated data annotation method for open-source data based on a multimodal large model, characterized in that, The open-source semi-automated data annotation method includes the following steps: Step S1: Perform enhancement processing on the original image dataset and the original text dataset in the open source data to obtain image-text data pairs; Step S2: Label the original image-text data pairs in the image-text data pair set with categories to determine the image-text-category data pair training set and the image-text-category data pair test set; Step S3: Input the image-text-category data pair training set into the SimCSE text encoder for training to adjust the SimCSE text encoder parameters. Step S4: Input the image-text-category data pair training set into the CLIP-SimCSE model, which includes the parameter-adjusted SimCSE text encoder and CLIP image encoder, and perform first image feature-similar text feature alignment, first image feature-image classification head alignment, and similar text feature-text classification head alignment respectively to adjust the CLIP-SimCSE model parameters. Step S5: Input the image-text-category data pair training set into the CLIP-ViT model, which includes the ViT image encoder and the CLIP text encoder, and perform second image feature-text feature alignment, second image feature-image classification head alignment, and text feature-text classification head alignment respectively to adjust the CLIP-ViT model parameters. Step S6: Input the image-text-category data pair test set into the multimodal large model including the CLIP-SimCSE model and the CLIP-ViT model after parameter adjustment, and conduct tests to adjust the parameters of the multimodal large model; Step S7: Input the unlabeled image and text data from the image-text data pair set into the multimodal large model after parameter adjustment, and perform category labeling to determine the different image-text data pairs in the labeling results of the CLIP-SimCSE model and the CLIP-ViT model; Step S8: Correct the different image-text data pairs in the annotation results.

2. The open-source semi-automated data annotation method according to claim 1, characterized in that, The specific implementation process of step S1 includes: Step S11: Rotate, crop, horizontally flip, and vertically flip each image in the original image dataset from the open-source data to form an enhanced image dataset; Step S12: Perform synonym replacement, random word insertion, and sentence rearrangement on each text in the original text dataset in the open source data to form an enhanced text dataset; Step S13: Perform image-text matching between the original image dataset and the original text dataset and the enhanced text dataset respectively, and perform image-text matching between the enhanced image data and the original text dataset and the enhanced text dataset respectively, to form an image-text data pair set.

3. The open-source semi-automated data annotation method according to claim 2, characterized in that, In step S3, the adjustment process of the SimCSE text encoder parameters includes: Step S31: Identify the positive sample sentences that are semantically similar to each sentence in the text of the image-text-category data training set and all the negative sample sentences that are semantically dissimilar. Step S32: Calculate the cosine similarity of each sentence with its corresponding positive sample sentence and all negative sample sentences to determine the loss value of each sentence; Step S33: Calculate the mean loss of the text using the loss value of each sentence; Step S34: Determine whether the mean loss of the text is less than the first threshold. If yes, end the training; otherwise, adjust the SimCSE text encoder parameters and return to step S31.

4. The open-source semi-automated data annotation method according to any one of claims 1 to 3, characterized in that, In step S4, the adjustment process of the CLIP-SimCSE model parameters includes: Step S41: Using the CLIP image encoder, extract image features from the image-text-category data training set to form the first image feature matrix; Step S42: Using the SimCSE text encoder, extract similar text features from the image-text-category data training set to form a similar text feature matrix; Step S43: Align the first image feature matrix and the similar text feature matrix with image features and similar text features to obtain the first loss value when aligning the image features and similar text features. Step S44: Align the first image feature matrix with the image classification head to obtain the second loss value when the first image feature matrix is ​​aligned with the image classification head; align the similar text feature matrix with the text classification head to obtain the third loss value when the similar text feature matrix is ​​aligned with the text classification head. Step S45: Calculate the first total loss value of the CLIP-SimCSE model based on the first loss value, the second loss value, and the third loss value; and determine whether the first total loss value is less than the second threshold. If yes, end the training; otherwise, adjust the CLIP-SimCSE model parameters and return to step S41.

5. The open-source semi-automated data annotation method according to claim 4, characterized in that, In step S5, the adjustment process of the CLIP-ViT model parameters includes: Step S51: Using the ViT image encoder, extract image features from each original image in the training set using image-text-category data to form a second image feature matrix that includes global image features and local image features of each corresponding important image patch. Step S52: Using the CLIP text encoder, extract text features from the image-text-category data training set to form a text feature matrix; Step S53: Perform global feature alignment and local feature alignment on the second image feature matrix and the text feature matrix respectively to obtain the fourth loss value for global feature alignment and the fifth loss value for local feature alignment; Step S54: Calculate the sixth loss value of the ViT image encoder based on the fourth and fifth loss values; and determine whether the sixth loss value is less than the third threshold. If yes, end the training and use the sixth loss value of the ViT image encoder as the sixth loss value obtained after training, and proceed to step S55; otherwise, adjust the parameters of the ViT image encoder and return to step S51. Step S55: Align the second image feature matrix with the image classification head to obtain the seventh loss value when the second image feature is aligned with the image classification head; align the text feature matrix with the text classification head respectively to obtain the eighth loss value when the text feature is aligned with the text classification head. Step S56: Calculate the second total loss value of the CLIP-ViT model based on the sixth, seventh, and eighth loss values ​​obtained after training; and determine whether the second total loss value is less than the fourth threshold. If yes, end the training; otherwise, adjust the CLIP-ViT model parameters and return to step S55.

6. The open-source semi-automated data annotation method according to claim 5, characterized in that, The specific implementation process of step S51 includes: Step S511: For the image-text-category data, each original image in the training set is sequentially divided into blocks and vectorized to form an image block vector sequence; Step S512: Add an image category to the beginning of the image block vector sequence to form a new vector sequence; Step S513: Add a positional code to the image category and each image block in the new vector sequence to form the image input sequence; Step S514: Input the image input sequence into the Transformer encoder to obtain global image features and local image features of each image patch; Step S515: Using global image features and local image features of each image patch, an attention mechanism is employed to calculate the contribution of each image patch to the global image features, in order to identify important image patches whose contribution is greater than the fifth threshold. Step S516: Use global image features and local image features of each important image block to form a second image feature matrix.

7. The open-source semi-automated data annotation method according to claim 6, characterized in that, In step S6, the testing process for the multimodal large model includes: Step S61: Using the CLIP-SimCSE model, after extracting image features and similar text features from the image-text-category data test set respectively, perform image feature-image classification head alignment and similar text feature-text classification head alignment respectively to determine the first test accuracy of the CLIP-SimCSE model. Step S62: Using the CLIP-ViT model, image features and text features are extracted from the image-text-category data of the test set, and then image feature-image classification head alignment and text feature-text classification head alignment are performed to determine the second test accuracy of the CLIP-ViT model. Step S63: Determine whether the accuracy of the first test and the accuracy of the second test are both greater than or equal to the sixth threshold. If yes, end the test; otherwise, adjust the model parameters for the model with an accuracy less than the sixth threshold and return to the corresponding model training process.

8. The open-source semi-automated data annotation method according to claim 7, characterized in that, The open-source semi-automated data annotation method also includes: Step S9: Add the corrected different image-text data pairs to the image-text-category data pair training set to update the image-text-category data pair training set.

9. A semi-automated data annotation system for open-source data based on a multimodal large model, characterized in that, The open-source semi-automated data annotation system includes: The enhancement processing module is used to perform enhancement processing on the original image dataset and the original text dataset in the open source data respectively to obtain image-text data pairs; The first category labeling module is used to label the original image-text data pairs in the image-text data pair set with categories in order to determine the image-text-category data pair training set and the image-text-category data pair test set; The first training module is used to input the image-text-category data pair training set into the SimCSE text encoder for training, so as to adjust the parameters of the SimCSE text encoder. The first alignment module is used to input the image-text-category data pair training set into the CLIP-SimCSE model, which includes the parameter-adjusted SimCSE text encoder and CLIP image encoder, and to perform first image feature-similar text feature alignment, first image feature-image classification head alignment, and similar text feature-text classification head alignment, respectively, so as to adjust the CLIP-SimCSE model parameters. The second alignment module is used to input the image-text-category data pair training set into the CLIP-ViT model, which includes the ViT image encoder and the CLIP text encoder, and to perform second image feature-text feature alignment, second image feature-image classification head alignment, and text feature-text classification head alignment, respectively, so as to adjust the CLIP-ViT model parameters. The testing module is used to input the image-text-category data pair test set into the multimodal large model, which includes the CLIP-SimCSE model and the CLIP-ViT model with parameter adjustments, respectively, for testing, so as to adjust the parameters of the multimodal large model; The second category labeling module is used to input unlabeled image and text data from the image-text data pair set into the parameter-adjusted multimodal large model for category labeling, so as to identify different image-text data pairs in the labeling results of the CLIP-SimCSE model and the CLIP-ViT model; The correction module is used to correct different image-text data pairs in the annotation results.

10. The open-source semi-automated data annotation system according to claim 9, characterized in that, The open-source semi-automated data annotation system also includes: The update module is used to add the corrected and different image-text data pairs to the image-text-class data pair training set to update the image-text-class data pair training set.

Images