Color selector image labeling model training method and device and storage medium

Abstract

The invention provides a color sorter image labeling model training method and device and a storage medium, and the method comprises the steps: obtaining an image category file containing category images and description information, generating image text pairs to form a training sample, constructing a label matrix representing the matching degree of the images and the texts through an initial labeling model, and obtaining a label matrix representing the matching degree of the images and the texts through the label matrix; and determining a loss value in combination with a preset loss function and adjusting model parameters to obtain an image annotation model. Therefore, the method can be applied to a scene in which multiple images correspond to the same text, does not need a large amount of manual annotation, reduces the data preparation and annotation cost, effectively reduces the domain difference between the color selector image and the pre-training model, and improves the annotation efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of color sorter image processing technology, and more specifically, to a color sorter image annotation model training method, device and storage medium. Background Technology

[0002] Color sorters are widely used for sorting agricultural and sideline products such as grains, nuts, and medicinal herbs. Their core functionality relies on high-precision image recognition for defect detection and product grading. With the increasing demand for quality inspection using artificial intelligence (AI), deep learning models have become the mainstream recognition solution, but their performance is highly limited by the quality and scale of the labeled data. Traditional debugging methods relying on human experience are no longer sufficient to meet the needs of rapid deployment of new product categories and real-time production line optimization.

[0003] Existing technologies mainly employ two methods. The first is manual annotation combined with supervised training using Convolutional Neural Networks (CNNs), where annotators label the image frame by frame with categories and defect types. The second is transfer learning based on visual language models such as Contrastive Language-Image Pre-training (CLIP). By freezing the text encoder and fine-tuning the visual encoder, the image is mapped to a joint embedding space aligned with the natural language description, and then zero-shot or few-shot recognition is achieved using cosine similarity.

[0004] However, in the first method, manual annotation is extremely inefficient, costly, and prone to introducing subjective bias. In the second method, the CLIP pre-training domain differs significantly from the color sorter image domain, and direct transfer can easily lead to distortion of feature representation. Furthermore, the loss function in the existing model only supports one-to-one matching, which can easily cause gradient conflicts, resulting in slow convergence and weak fine-grained discrimination ability. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a method, device, and storage medium for training an image annotation model for a color sorter, thereby solving the problems of low efficiency and low accuracy in image annotation in the prior art.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a method for training an image annotation model for a color sorter, the method comprising: Obtain multiple image category files, each of which includes multiple images of one category and description information of the category; Multiple image-text pairs are generated based on the multiple image category files to form training samples, wherein each image-text pair includes a color sorter material image and descriptive text of the color sorter material image; The training samples are input into the initial annotation model, and the initial annotation model constructs a label matrix corresponding to the training samples. The values ​​of each matrix element in the label matrix are used to determine the matching degree between each image and each descriptive text. The loss value of the initial annotation model is determined based on the label matrix and the preset loss function, and the parameters of the initial annotation model are adjusted based on the loss value to obtain the image annotation model.

[0007] Optionally, generating multiple image-text pairs based on the multiple image category files to form training samples includes: The category description information in the image category file is input into a pre-trained generative model, which performs semantic expansion on the description information to obtain multiple descriptive texts corresponding to the description information. Each image in the image category file is combined with its corresponding descriptive text to obtain the image-text pair, which constitutes the training sample.

[0008] Optionally, the initial annotation model includes: a text encoder; The process of constructing the label matrix corresponding to the training samples using the initial annotation model includes: Traverse each image, and for the current image that has been traversed, encode the descriptive text of the current image according to the text encoder to obtain the first text feature; Each description text is traversed, and for the current description text that is traversed, the current description text is encoded according to the text encoder to obtain the second text feature; The similarity between the first text feature and the second text feature is calculated to obtain the similarity between the image and the current description text, and the values ​​of the matrix elements corresponding to the image and the current description text are determined based on the similarity.

[0009] Optionally, determining the values ​​of the matrix elements corresponding to the image and the current descriptive text based on the similarity includes: If the similarity is greater than a preset threshold, then the value of the matrix element corresponding to the image and the current description text is determined to be a first value, and the first value indicates that the image and the current description text match. If the similarity is less than or equal to a preset threshold, then the value of the matrix element corresponding to the image and the current description text is determined to be a second value, which indicates that the image and the current description text do not match.

[0010] Optionally, determining the loss value of the initial annotation model based on the label matrix and a preset loss function includes: Traverse each element in the label matrix, and for the current element, determine the loss value of the current element according to the loss function. The loss value of the initial labeling model is determined based on the loss value of each matrix element in the label matrix.

[0011] Optionally, the initial annotation model includes: a visual encoder; The step of adjusting the parameters of the initial annotation model based on the loss value to obtain the image annotation model includes: The parameters of the visual encoder are adjusted based on the loss value to obtain the image annotation model.

[0012] Optionally, the method further includes: The target description text is input into the image annotation model, and the image annotation model encodes the target description text to obtain the first text feature corresponding to the target description text; Each of the images to be labeled is encoded to obtain the first image features of each of the images to be labeled. Based on the first text feature and the first image feature of each of the images to be labeled, the target labeled image corresponding to the target description text is determined in each of the images to be labeled.

[0013] Optionally, the method further includes: Multiple support set images are input into the image annotation model, and the image annotation model encodes each support set image to obtain the second image feature corresponding to each support set image. Each of the images to be labeled is encoded to obtain the third image features of each of the images to be labeled; Based on the second image features of each support set image and the third image features of each image to be labeled, the target labeled image corresponding to the support set image is determined in each image to be labeled.

[0014] Secondly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of a color sorter image annotation model training method as described in any one of the first aspects.

[0015] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of a color sorter image annotation model training method as described in any one of the first aspects.

[0016] The beneficial effects of this application are: by integrating category images and descriptive information to generate training samples, it can adapt to scenarios where multiple images correspond to the same text, thereby improving data utilization. By quantifying the matching relationship between images and text through a label matrix, calculating the loss value by combining a loss function, and adjusting the model parameters in a targeted manner, the domain difference between color sorter images and pre-trained models can be effectively reduced. The resulting image annotation model does not require a large amount of manual annotation, reducing data preparation costs and annotation difficulty, and can significantly improve the efficiency and accuracy of color sorter image annotation.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a color sorter image annotation model training method provided in an embodiment of this application is shown; Figure 2 This document illustrates a flowchart of a method for generating training samples according to an embodiment of this application. Figure 3 This document illustrates a flowchart of a tag matrix generation method provided in an embodiment of this application. Figure 4 A flowchart illustrating a method for determining the numerical values ​​of matrix elements according to an embodiment of this application is shown; Figure 5 This application provides a flowchart for determining a loss value according to an embodiment of the present application. Figure 6 This document illustrates a flowchart of a method for generating a target-annotated image, as provided in an embodiment of this application. Figure 7 This document illustrates a flowchart of another method for generating a target-annotated image, as provided in an embodiment of this application. Figure 8 This illustration shows a schematic diagram of the structure of a color sorter image annotation model training device provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Currently, image recognition in color sorters mainly relies on manual annotation combined with deep learning models, or directly transferring the CLIP model based on contrastive learning. This model consists of a visual encoder and a text encoder, and achieves cross-modal matching by the similarity between images and text in a unified feature space, demonstrating strong generalization ability in zero-shot and few-shot tasks.

[0023] However, in existing technologies, manual annotation is inefficient and costly, making it difficult to support rapid debugging and large-scale application of color sorters. In the second approach, the CLIP model is pre-trained on general image data, which differs significantly from the agricultural product images collected by color sorters in terms of imaging conditions, target scale, and background complexity, resulting in poor direct transfer performance. Furthermore, existing technologies lack image-text pairing datasets specifically for color sorters, making construction difficult and time-consuming. In addition, mainstream loss functions only support one-to-one image-text matching, failing to handle common situations where multiple images correspond to the same type of text description, such as multiple normal peanut images matching complete shapes and light brown natural language descriptions. This leads to unstable training, slow convergence, and weak fine-grained discrimination ability.

[0024] Based on this, this application proposes a training method for an image annotation model of a color sorter. By acquiring image category files containing category images and descriptive information, image-text pairs are generated to form training samples. An initial annotation model is used to construct a label matrix representing the degree of matching between images and text. A preset loss function is used to determine the loss value and adjust the model parameters to obtain the image annotation model. This method can then be applied to scenarios where multiple images correspond to the same text, eliminating the need for extensive manual annotation, reducing data preparation and annotation costs, effectively narrowing the domain differences between color sorter images and pre-trained models, and improving annotation efficiency and accuracy.

[0025] The following describes the image annotation model training method for color sorters in this application. The subject executing this method can be an electronic device, such as... Figure 1 As shown, the method includes: S101. Obtain multiple image category files, each image category file including multiple images of one category and category description information.

[0026] Optionally, material images from the same type of color sorter can be stored in the same folder to obtain an image category file. An image category file is a file carrier used to store data related to materials from the same category of color sorter. It contains multiple material images under that category and information describing the core characteristics of that category. For example, a "normal Brazilian pine nuts" category file contains multiple images of Brazilian pine nuts with intact shells. The folder name of the category file can serve as the category's descriptive information, or simple descriptive information can be added to the category file itself. For example, the descriptive information for the "normal Brazilian pine nuts" category file could be "intact shells, no damage or cracks."

[0027] In one possible implementation, material images from a color sorter can be collected, binarized, the foreground of the material can be segmented and the bounding rectangle extracted, individual material images can be cropped, and then the images can be classified and stored in corresponding folders according to their categories to form an image collection. Finally, basic description information can be written for each category to form a complete image category file.

[0028] S102. Generate multiple image-text pairs based on multiple image category files to form training samples. Each image-text pair includes a color sorter material image and a descriptive text for the color sorter material image.

[0029] Optionally, the image-text pair can be a data pair consisting of a single color sorter material image and its corresponding descriptive text, which is the basic unit for model training. The descriptive text can accurately reflect the category or surface features of the material in the image.

[0030] In one possible implementation, the category description information can be extended by generating multiple description texts for each image category file, and each description text can be combined with an image in the image category file to obtain an image-text pair.

[0031] For example, assuming the image category file contains 5 images, and the category description information is expanded to obtain 2 description texts, then each image can be combined with the description text to obtain 10 image-text pairs as training samples.

[0032] By associating images with text to form training samples, the model can learn the correspondence between image features and text descriptions, providing structured data for subsequent model training. It can also support scenarios where multiple images correspond to the same text, improving data utilization.

[0033] S103. Input the training samples into the initial annotation model, and construct the label matrix corresponding to the training samples from the initial annotation model. The values ​​of each matrix element in the label matrix are used to determine the matching degree of each image and each descriptive text.

[0034] The initial annotation model can be a base model built on a pre-trained CLIP model, which includes a visual encoder and a text encoder, used to initially process training samples and build a label matrix.

[0035] Optionally, the label matrix can be an N×N matrix (where N is the number of image-text pairs), and the matrix element values ​​are used to represent the matching relationship between the image and the text at the corresponding location.

[0036] It should be noted that during the initial annotation model training process, all weight parameters of the text encoder can be frozen, and only the parameters of the visual encoder in the model can be adjusted. This can improve the model's attention to the features of the color sorter image during the model training process and avoid destroying the existing semantic space structure of the descriptive text.

[0037] In one possible implementation, after inputting the training samples into the initial annotation model, the model's text encoder encodes the descriptive text in each image-text pair to obtain the corresponding text features. For each image, all descriptive texts are traversed, and the similarity between the text features corresponding to that image and the current descriptive text features is calculated. If the similarity is greater than a preset threshold, the corresponding element in the label matrix is ​​set to +1 (representation matching); otherwise, it is set to -1 (representation mismatch), thus forming a complete label matrix.

[0038] S104. Determine the loss value of the initial annotation model based on the label matrix and the preset loss function, and adjust the parameters of the initial annotation model according to the loss value to obtain the image annotation model.

[0039] The loss function is used to measure the deviation between the model's prediction and the actual matching relationship. For example, it can be the MultiMatch-SigLipLoss loss function.

[0040] Optionally, the deviation between the model's predicted results and the ideal results can be calculated using a loss function. The model parameters can then be adjusted by backpropagation based on the loss value, allowing the model to gradually adapt to the image features of the color sorter and improve the labeling accuracy.

[0041] It is worth noting that the weight parameters of the text encoder are frozen during model training. When adjusting the parameters of the initial annotation model based on the loss value, the weight parameters of the visual encoder in the initial annotation model can be adjusted to obtain the image annotation model.

[0042] In this embodiment, training samples are generated by integrating category images and descriptive information, which can adapt to scenarios where multiple images correspond to the same text, thereby improving data utilization. The matching relationship between images and text is quantified by the label matrix, the loss value is calculated by combining the loss function, and the model parameters are adjusted in a targeted manner to effectively reduce the domain difference between color sorter images and pre-trained models. The resulting image annotation model does not require a large amount of manual annotation, which reduces the data preparation cost and annotation difficulty, and can significantly improve the efficiency and accuracy of color sorter image annotation.

[0043] The following is a further explanation of the above method of generating multiple image-text pairs based on multiple image category files to form training samples, such as... Figure 2 As shown, the above step S102 includes: S201. Input the category description information from the image category file into the pre-trained generative model. The generative model performs semantic expansion on the description information to obtain multiple description texts corresponding to the description information.

[0044] Generative models can be artificial intelligence models with semantic expansion and text generation capabilities. They can generate diverse and linguistically logical extended texts based on the basic descriptive information of the input, thereby enhancing the richness of language supervision.

[0045] It should be understood that the original category description information is usually concise and lacks sufficient linguistic supervision information. By using generative models to extend semantics, the diversity and detail of the text can be enriched, enabling the model to learn the multi-dimensional feature associations of materials in the same category, improving the model's ability to distinguish fine-grained categories, and adapting to scenarios where multiple images correspond to the same text.

[0046] In one possible implementation, basic description information of the category can be extracted from the image category file first. This description information is then input into a pre-trained generative model, semantic extension parameters are set, and multiple descriptive texts that retain the core meaning but express different meanings are generated. Finally, effective extended texts are selected to form a set of descriptive texts corresponding to the category.

[0047] S202. Combine each image in the image category file with its corresponding descriptive text to obtain image-text pairs, which constitute training samples.

[0048] Optionally, images of the same category can be combined one by one with all the descriptive texts corresponding to that category to construct a sufficient and diverse number of image-text pairs. This allows the training samples to fully cover the association between category features and textual descriptions, providing ample data for the model to learn the matching relationship between images and texts, while also being compatible with the situation where multiple images correspond to the same text.

[0049] In one possible implementation, each material image in the image category file can be traversed, and each image can be paired with all the descriptive texts under that category to form multiple image-text pairs. For example, an image of "normal peanuts" can be combined with descriptive texts such as "full kernels, uniform light red seed coat, and intact shape" and "intact shape, uniform light brown color". All image-text pairs corresponding to the categories are collected and organized according to a preset format to form a complete training sample.

[0050] In this embodiment, the semantic extension of category description information is performed by generating a model, which enriches the text supervision information and improves the model's ability to distinguish fine-grained categories. By combining images with multiple descriptive texts to construct training samples, existing classification datasets can be used efficiently, reducing data preparation costs. At the same time, it adapts to scenarios where multiple images correspond to the same text. The final training sample data is sufficient and highly diverse, providing a high-quality training foundation for the initial annotation model, thereby improving the model's annotation accuracy and generalization ability.

[0051] Optionally, the initial annotation model includes a text encoder. The text encoder is used to convert natural language descriptive text into computer-recognizable feature vectors.

[0052] like Figure 3 As shown, the process of constructing the label matrix corresponding to the training samples using the initial annotation model includes: S301. Traverse each image. For the current image that has been traversed, encode the description text of the current image according to the text encoder to obtain the first text feature.

[0053] The first text feature is a feature vector obtained by encoding the descriptive text corresponding to the current image through a text encoder. It is used to reflect the core text features of the category to which the image belongs.

[0054] In one possible implementation, the text encoder in the initial annotation model can be started and its parameters frozen. Then, each image in the training samples is traversed to determine the descriptive text corresponding to the current image, and the descriptive text is input into the text encoder. After encoding, the corresponding first text feature is output.

[0055] S302. Traverse each description text. For the current description text that has been traversed, encode the current description text according to the text encoder to obtain the second text feature.

[0056] The second text feature is a feature vector obtained by encoding the descriptive text in the training samples using a text encoder, and is used to compare its similarity with the first text feature.

[0057] In one possible implementation, the descriptive text corresponding to all image-text pairs in the training samples can be traversed, and for each current descriptive text traversed, it can be input into the text encoder with frozen parameters, and the corresponding second text feature can be obtained after encoding processing.

[0058] S303. Calculate the similarity between the first text feature and the second text feature to obtain the similarity between the image and the current description text, and determine the values ​​of the matrix elements corresponding to the image and the current description text based on the similarity.

[0059] The semantic association between the first and second text features is quantified by similarity calculation. The similarity level reflects the degree of matching between the image and the descriptive text. Based on this similarity, the judgment criteria can be set to accurately determine the value of the corresponding element in the label matrix, providing a reliable label for subsequent loss calculation.

[0060] In one possible implementation, a cosine similarity algorithm can be used to calculate the similarity between the first text feature of the current image and each second text feature, and determine the values ​​of the matrix elements corresponding to the image and the current descriptive text based on the similarity.

[0061] The label matrix is ​​an N×N binary matrix, where N is the number of image-text pairs in the training samples. Each element in the matrix has a value of +1 or -1, used to explicitly represent the matching relationship between the color sorter material image and the descriptive text at the corresponding location. In one possible implementation, when filling the label matrix based on the matching results, the rows of the matrix correspond to images, and the columns correspond to descriptive text. If the image in the i-th row and j-th column matches the descriptive text, then Y... ij =+1; if not a match, then Y ij =-1. For example, the row corresponding to the image of "normal peanuts" has elements of +1 in the columns corresponding to all descriptive text related to "normal peanuts", while the columns corresponding to descriptive text related to "peeled peanuts" and "cracked Brazilian pine nuts" have elements of -1.

[0062] It should be noted that in the training samples for color sorter image annotation, there may be multiple images corresponding to the same text. Existing label matrices provide a one-to-one match, which cannot accurately reflect this correspondence in the color sorter image annotation scenario. The label matrix in this application, however, can reflect this many-to-one matching relationship between images and descriptive text, thus providing accurate labels for the loss function and ensuring that the model can learn the association between images and text within the same category.

[0063] In this embodiment, the descriptive text is standardized and encoded using a text encoder to obtain text features in a unified feature space, ensuring the accuracy of similarity calculation. Based on the similarity measure, the matching relationship between the image and the descriptive text is standardized, and a label matrix is ​​constructed. This provides a reliable basis for model loss calculation, thereby adapting to scenarios where multiple images correspond to the same text, improving the initial annotation model's learning effect on the association between color sorter images and text, and enhancing the accuracy of subsequent image annotation.

[0064] The following is a further explanation of the matrix element values ​​for determining the image and the current descriptive text based on similarity, such as... Figure 4 As shown, the above step S303 includes: S401. If the similarity is greater than a preset threshold, the value of the matrix element corresponding to the image and the current description text is determined to be the first value, and the first value represents the matching between the image and the current description text.

[0065] The preset threshold refers to a pre-defined critical value used to determine whether the text feature similarity meets the standard. Similarity reflects the semantic correlation between the first text feature corresponding to the image and the second text feature of the current descriptive text. When the similarity exceeds the preset threshold, it indicates that the core features of the two are consistent, that is, the image and the descriptive text belong to the same category. The first numerical value can clearly identify the matching relationship and provide a supervision signal for the model to learn the common features under the same category.

[0066] In one possible implementation, a reasonable preset threshold (such as 0.8, as verified by experiments) can be pre-set based on the characteristics of the color sorter image dataset. The cosine similarity between the first text feature of the current image and the second text feature of the current descriptive text is then calculated. If the calculated result is greater than the preset threshold, the image is determined to match the current descriptive text, and the matrix element value at the corresponding position in the label matrix is ​​set to the first value, which could be, for example, +1. For instance, if the similarity between the first text feature of the image "normal peanuts" and the second text feature of the descriptive text "uniform light brown" is 0.85, which is greater than the preset threshold of 0.8, then the matrix element values ​​corresponding to image A and the descriptive text can be set to +1.

[0067] S402. If the similarity is less than or equal to a preset threshold, the value of the matrix element corresponding to the image and the current description text is determined to be the second value, which indicates that the image and the current description text do not match.

[0068] In one possible implementation, after similarity calculation, if the result is less than or equal to a preset threshold, the image is determined to be mismatched with the current descriptive text. The corresponding matrix element value in the label matrix is ​​then set to a second value, which could be, for example, -1. For instance, the first text feature of the image "normal peanuts" has a similarity of 0.6 to the second text feature of the descriptive text "uneven light brown," which is less than the preset threshold of 0.8; therefore, the corresponding matrix element value is set to -1.

[0069] As one possible implementation, the values ​​of each element in the label matrix can be determined using the following equation (1). This represents the matrix element in the i-th row and j-th column. Descriptive text representing the image. This represents the current description text. If the image's description text is the same as or similar to the current description text, the value of the matrix element is +1; if the image's description text is different from or dissimilar to the current description text, the value of the matrix element is -1.

[0070] (1) In this embodiment, a preset threshold is used to define the matching criteria between the image and the descriptive text. The first and second numerical values ​​are used to accurately identify the matching and non-matching relationships, thereby ensuring that the label matrix can objectively reflect the association between multiple images and the same text in the training samples, and providing accurate label basis for the loss function.

[0071] The following is a further explanation of how the loss value of the initial labeled model is determined based on the label matrix and the preset loss function. Figure 5 As shown, step S104 above includes: S501. Traverse each element in the label matrix. For the current element, determine the loss value based on the loss function.

[0072] In this context, a matrix element refers to a specific unit in the label matrix. Each matrix element corresponds to a matching relationship between a color sorter material image and a descriptive text. A matrix element with a value of the first value indicates that the image and the descriptive text match, while a value of the second value indicates that the image and the descriptive text do not match.

[0073] Each element in the label matrix corresponds to a true matching relationship of a set of image-text pairs. By calculating the loss value of each element one by one through the loss function, the prediction deviation of each set of image-text pairs can be accurately located, providing a basis for subsequent model parameter adjustment, and thus adapting to the scenario where multiple images correspond to the same text.

[0074] In one possible implementation, after activating the preset MultiMatch-SigLipLoss loss function, all matrix elements in the label matrix can be traversed in row or column order. For the current matrix element, its corresponding image feature vector and text feature vector are obtained, their similarity score is calculated, and then substituted into the loss function formula to obtain the loss value of the current matrix element. For example, if the current matrix element is +1 (image and text matching), and the similarity score is low, the calculated loss value will be large, reflecting a significant prediction deviation in this matching relationship.

[0075] S502. Determine the loss value of the initial annotation model based on the loss value of each matrix element in the label matrix.

[0076] Optionally, after calculating the loss values ​​for all matrix elements, the total loss value of the initial labeled model is calculated using an average summation method. For example, if the label matrix contains 128 elements (N=128) and the sum of the loss values ​​of all elements is 64, then the total loss value of the model is 64÷128=0.5, which can reflect the current overall prediction bias level of the model.

[0077] As one possible implementation, the loss function can be shown in equation (2).

[0078] (2) in, This is the current matrix element value. The similarity score is represented by σ, which is the logsigmoid function.

[0079] In this embodiment, the loss value is calculated element by element to accurately capture the prediction deviation of each image-text pair. This can be adapted to scenarios where multiple images correspond to the same text. By summarizing the results, the total loss value of the model can be obtained, which can comprehensively reflect the overall prediction level of the model.

[0080] Optionally, the initial annotation model includes a visual encoder. The visual encoder is used to convert the material image from the color sorter into a computer-recognizable feature vector. The process of adjusting the parameters of the initial annotation model based on the loss value to obtain the image annotation model includes: The parameters of the visual encoder are adjusted based on the loss value to obtain the image annotation model.

[0081] The initial visual encoder is trained on general pre-trained data, which differs from the color sorter images. Therefore, its parameters cannot accurately extract the material features of the color sorter. The loss value reflects the model's prediction bias. Therefore, adjusting the visual encoder parameters based on the loss value allows the model to gradually learn the specific features of the color sorter images, reducing domain differences and improving annotation adaptability.

[0082] In one possible implementation, the parameters of the text encoder in the initial annotation model can be frozen, and the calculated total loss value of the model can be passed to the visual encoder through the backpropagation algorithm. The AdamW optimizer, combined with the preset initial learning rate and learning rate scheduler, is used to iteratively adjust the weight parameters of the visual encoder. After a preset number of training rounds, until the loss function converges, an image annotation model adapted to the color sorter image is obtained.

[0083] In this embodiment, by freezing the text encoder parameters and adjusting the visual encoder parameters only, the image features of the color sorter can be accurately adapted, effectively reducing regional differences.

[0084] After obtaining the image annotation model, the recognition and annotation of material images from color sorters can be achieved based on the image annotation model, such as... Figure 6 As shown, the method of this application further includes: S601. Input the target description text into the image annotation model, and have the image annotation model encode the target description text to obtain the first text feature corresponding to the target description text.

[0085] In this context, the target description text is natural language text input by the annotator to specify the features of the category to be labeled. The image annotation model can filter and obtain the target labeled images from the images to be labeled based on the target description text.

[0086] The text encoder in the image annotation model can encode the target description text to obtain the first text feature corresponding to the target description text.

[0087] S602. Encode each image to be labeled to obtain the first image feature of each image to be labeled.

[0088] In one possible implementation, images of materials to be labeled in a color sorter can be collected, and the images can be preprocessed such as binarization and cropping. Each preprocessed image to be labeled is then input into the visual encoder of the image labeling model. The visual encoder extracts features from the images to be labeled to obtain the first image features corresponding to each image.

[0089] S603. Based on the first text features and the first image features of each image to be labeled, determine the target labeled image corresponding to the target description text in each image to be labeled.

[0090] Optionally, a cosine similarity algorithm can be used to calculate the similarity between the first text feature and the first image feature of each image to be labeled, set similarity ranking rules, select the image to be labeled with the highest similarity, determine the image as the target labeled image corresponding to the target description text, and assign the corresponding category label.

[0091] For example, annotators can input target descriptive text into the image annotation model, such as the prompts describing normal peanuts: "full kernels, uniform light red or light brown seed coat, intact shape, elongated oval, no cracks, no breakage, no insect infestation, no mold spots." The image to be annotated is also input into the image annotation model. The image annotation model can encode the target descriptive text to obtain the first text feature, encode the image to be annotated to obtain the first image feature, and calculate the similarity between the first text feature and the first image feature. If the similarity between the first text feature and the first image feature meets the preset requirements, the annotation information "normal peanut" can be added to the image to be annotated, and the image to be annotated with the added annotation information can be used as the target annotation image.

[0092] In this embodiment, the target description text is converted into quantitative features through text encoding. Combined with the features of the image to be labeled extracted by the fine-tuned visual encoder, the target labeled image is accurately screened through similarity matching. This adapts to the rapid initial screening scenario of new defects and new categories, and can significantly reduce the cost of manual labeling, improve the flexibility and efficiency of color sorter image labeling, while ensuring the accuracy of labeling.

[0093] In the second implementation method, to improve the accuracy of image annotation, image annotation can also be performed based on a small number of sample images, such as... Figure 7 As shown, the method of this application further includes: S701. Input multiple support set images into the image annotation model, and have the image annotation model encode each support set image to obtain the second image feature corresponding to each support set image.

[0094] Among them, the support set image refers to a small number of color sorter material images that are manually labeled by the labelers and have category representativeness. For example, images with labeling information of different types such as "normal peanuts", "damaged peanuts", and "insect-damaged peanuts" can be used as support set images and input into the image labeling model.

[0095] The image annotation model can encode images in each support set to obtain the second image features of each image, and establish a feature retrieval library based on the second image features. The feature retrieval library includes multiple second image features, and each second image feature has corresponding annotation information.

[0096] S702. Encode each image to be labeled to obtain the third image features of each image to be labeled.

[0097] Optionally, images of materials to be labeled in a color sorter can be collected, processed according to the preprocessing standards of the training data, and each preprocessed image to be labeled can be input into the visual encoder of the image labeling model. The image labeling model can then extract features from the images to be labeled to obtain the third image features corresponding to each image to be labeled.

[0098] S703. Based on the second image features of each support set image and the third image features of each image to be labeled, determine the target labeled image corresponding to the support set image in each image to be labeled.

[0099] Optionally, the second image features of all support set images can be aggregated to construct a feature retrieval library. Then, the cosine similarity algorithm is used to calculate the similarity between the third image feature of each image to be labeled and all the second image features in the retrieval library. Finally, the support set image category corresponding to the second image feature with the highest similarity is selected and used as the label of the image to be labeled, thus determining it as the target labeled image.

[0100] For example, annotators can input a small number of support set images with different annotation information into the image annotation model, and also input the image to be annotated into the image annotation model. The image annotation model can encode the support set images to obtain second image features, and establish a feature retrieval library corresponding to each annotation information based on the second image features. The image to be annotated is then encoded to obtain third image features. The third image features and the second image features of all support set images in the feature retrieval library are used to calculate cosine similarity. The annotation information of the support set image most similar to the image to be annotated is used as the annotation information of the image to be annotated. Annotation information is added to the image to be annotated, and the image to be annotated with the added annotation information is used as the target annotation image.

[0101] In this embodiment, only a small number of manually labeled support set images are needed to build a feature retrieval library, which greatly reduces labor costs. By fine-tuning the accurate features extracted by the visual encoder and classifying the images to be labeled through similarity matching, the accuracy of labeling can be improved in small sample scenarios.

[0102] Based on the same inventive concept, this application also provides a color sorter image annotation model training device corresponding to the color sorter image annotation model training method. Since the principle of the device in this application is similar to the color sorter image annotation model training method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0103] Figure 8A schematic diagram of the structure of a color sorter image annotation device provided in an embodiment of this application is shown.

[0104] The acquisition module 801 is used to acquire multiple image category files, each image category file including multiple images of one category and category description information; The generation module 802 is used to generate multiple image-text pairs based on multiple image category files to form training samples. Each image-text pair includes a color sorter material image and descriptive text of the color sorter material image. The construction module 803 is used to input training samples into the initial annotation model, and the initial annotation model constructs the label matrix corresponding to the training samples. The values ​​of each matrix element in the label matrix are used to determine the matching degree of each image and each descriptive text. Training module 804 is used to determine the loss value of the initial annotation model based on the label matrix and the preset loss function, and to adjust the parameters of the initial annotation model according to the loss value to obtain the image annotation model.

[0105] Optionally, the generation module 802 is specifically used for: The category description information in the image category file is input into the pre-trained generative model, which performs semantic expansion on the description information to obtain multiple description texts corresponding to the description information. Each image in the image category file is combined with its corresponding descriptive text to obtain image-text pairs, which constitute training samples.

[0106] Optionally, the initial annotation model includes: a text encoder; Optionally, module 803 is specifically used for: Traverse each image, and for the current image, encode the descriptive text of the current image according to the text encoder to obtain the first text feature; Iterate through each description text, and for the current description text that has been traversed, encode the current description text according to the text encoder to obtain the second text feature; The similarity between the first text feature and the second text feature is calculated to obtain the similarity between the image and the current description text, and the values ​​of the matrix elements corresponding to the image and the current description text are determined based on the similarity.

[0107] Optionally, module 803 is specifically used for: If the similarity is greater than a preset threshold, the value of the matrix element corresponding to the image and the current description text is determined to be the first value, which indicates that the image and the current description text match. If the similarity is less than or equal to a preset threshold, the value of the matrix element corresponding to the image and the current description text is determined to be the second value, which indicates that the image and the current description text do not match.

[0108] Optionally, training module 804 is specifically used for: Iterate through each element in the label matrix, and for the current element, determine the loss value of the current element based on the loss function. The loss value of the initial labeling model is determined based on the loss value of each matrix element in the label matrix.

[0109] Optionally, the initial annotation model includes: a visual encoder; Training module 804 is specifically used for: The parameters of the visual encoder are adjusted based on the loss value to obtain the image annotation model.

[0110] Optionally, the device also includes a labeling module, specifically used for: The target description text is input into the image annotation model, which encodes the target description text to obtain the first text feature corresponding to the target description text. Each image to be labeled is encoded to obtain the first image feature of each image; Based on the first text features and the first image features of each image to be labeled, the target labeled image corresponding to the target description text is determined in each image to be labeled.

[0111] Optionally, the annotation module is specifically used for: Multiple support set images are input into the image annotation model, and the image annotation model encodes each support set image to obtain the second image feature corresponding to each support set image. Each image to be labeled is encoded to obtain the third image features of each image; Based on the second image features of each support set image and the third image features of each image to be labeled, the target labeled image corresponding to the support set image is determined in each image to be labeled.

[0112] This application's embodiments generate training samples by integrating category images and descriptive information, which can adapt to scenarios where multiple images correspond to the same text, improving data utilization. The matching relationship between images and text is quantified through a label matrix, and the loss value is calculated by combining a loss function. The model parameters are adjusted in a targeted manner, effectively reducing the domain difference between color sorter images and pre-trained models. The resulting image annotation model does not require a large amount of manual annotation, reducing data preparation costs and annotation difficulty, and can significantly improve the efficiency and accuracy of color sorter image annotation.

[0113] Figure 9This illustration shows a schematic diagram of an electronic device provided in an embodiment of this application, including: a processor 901, a storage medium 902, and a bus 903. The storage medium 902 stores machine-readable instructions executable by the processor 901. When the electronic device runs a color sorter image annotation model training method as described in the embodiment, the processor 901 communicates with the storage medium 902 via the bus 903. The processor 901 executes the machine-readable instructions. The preamble of the method item of the processor 901 executes the steps in the above-described color sorter image annotation model training method.

[0114] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor, which performs the steps in the above-described color sorter image annotation model training method.

[0115] In this embodiment, the computer program, when run by the processor, can also execute other machine-readable instructions to perform other methods as described in the embodiments. For details on the specific execution steps and principles, please refer to the description of the embodiments, which will not be repeated here.

[0116] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0121] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for training an image annotation model for a color sorter, characterized in that, include: Obtain multiple image category files, each of which includes multiple images of one category and description information of the category; Multiple image-text pairs are generated based on the multiple image category files to form training samples, wherein each image-text pair includes a color sorter material image and descriptive text of the color sorter material image; The training samples are input into the initial annotation model, and the initial annotation model constructs a label matrix corresponding to the training samples. The values ​​of each matrix element in the label matrix are used to determine the matching degree between each image and each descriptive text. The loss value of the initial annotation model is determined based on the label matrix and the preset loss function, and the parameters of the initial annotation model are adjusted based on the loss value to obtain the image annotation model.

2. The method according to claim 1, characterized in that, The step of generating multiple image-text pairs based on the multiple image category files to form training samples includes: The category description information in the image category file is input into a pre-trained generative model, which performs semantic expansion on the description information to obtain multiple descriptive texts corresponding to the description information. Each image in the image category file is combined with its corresponding descriptive text to obtain the image-text pair, which constitutes the training sample.

3. The method according to claim 1, characterized in that, The initial annotation model includes: a text encoder; The process of constructing the label matrix corresponding to the training samples using the initial annotation model includes: Traverse each image, and for the current image that has been traversed, encode the descriptive text of the current image according to the text encoder to obtain the first text feature; Each description text is traversed, and for the current description text that is traversed, the current description text is encoded according to the text encoder to obtain the second text feature; The similarity between the first text feature and the second text feature is calculated to obtain the similarity between the image and the current description text, and the values ​​of the matrix elements corresponding to the image and the current description text are determined based on the similarity.

4. The method according to claim 3, characterized in that, Determining the values ​​of the matrix elements corresponding to the image and the current descriptive text based on the similarity includes: If the similarity is greater than a preset threshold, then the value of the matrix element corresponding to the image and the current description text is determined to be a first value, and the first value indicates that the image and the current description text match. If the similarity is less than or equal to a preset threshold, then the value of the matrix element corresponding to the image and the current description text is determined to be a second value, which indicates that the image and the current description text do not match.

5. The method according to claim 1, characterized in that, The step of determining the loss value of the initial labeling model based on the label matrix and a preset loss function includes: Iterate through each element in the label matrix, and for the current element, determine the loss value of the current element according to the loss function. The loss value of the initial labeling model is determined based on the loss value of each matrix element in the label matrix.

6. The method according to claim 1, characterized in that, The initial annotation model includes: a visual encoder; The step of adjusting the parameters of the initial annotation model based on the loss value to obtain the image annotation model includes: The parameters of the visual encoder are adjusted based on the loss value to obtain the image annotation model.

7. The method according to claim 1, characterized in that, The method further includes: The target description text is input into the image annotation model, and the image annotation model encodes the target description text to obtain the first text feature corresponding to the target description text; Each image to be labeled is encoded to obtain the first image feature of each image to be labeled. Based on the first text feature and the first image feature of each of the images to be labeled, the target labeled image corresponding to the target description text is determined in each of the images to be labeled.

8. The method according to claim 1, characterized in that, The method further includes: Multiple support set images are input into the image annotation model, and the image annotation model encodes each support set image to obtain the second image feature corresponding to each support set image. The images to be labeled are encoded to obtain the third image features of each image to be labeled; Based on the second image features of each support set image and the third image features of each image to be labeled, the target labeled image corresponding to the support set image is determined in each image to be labeled.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of a color sorter image annotation model training method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the color sorter image annotation model training method as described in any one of claims 1 to 8.

Images