Model training method and device, electronic equipment and storage medium

Abstract

The invention provides a model training method and device, electronic equipment and a storage medium. The training method comprises the following steps: for a first image, determining a second image corresponding to the first image, and determining a first positive example feature distance between a first feature corresponding to the first image and a second feature corresponding to the second image; for each first image, determining a plurality of different-source images, respectively determining a first feature distance between the first feature and a different-source feature corresponding to each different-source image, and obtaining a first negative example feature distance corresponding to the first image; determining a first loss value of the model according to the first positive example feature distance and the first negative example feature distance corresponding to each first image; determining a final loss value of the model according to the first loss value; and training the model by using the final loss value. According to the scheme, the model is trained through the first loss value capable of representing the feature distance, so that the classification precision of the model for the to-be-detected image can be improved, and the practicability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a model training method, a model training device, an electronic device, a storage medium and a computer program product. Background Art

[0002] In the industrial field, the popularization of artificial intelligence technology has effectively reduced labor costs. The core of artificial intelligence technology is the training of machine learning models. Training machine learning models in semi-supervised learning mode usually results in low model accuracy.

[0003] For example, for a learning model used for category recognition, it can be trained using a semi-supervised learning mode or an unsupervised learning mode, which can not only recognize known categories, but also recognize potential new categories in unlabeled data. However, in the related prior art, a contrastive learning method is usually used to pseudo-label the unlabeled data, convert the open classification task into a closed-set classification task, and then directly use the classification results of the labeled data to train the model, which cannot meet the user's accuracy requirements for the classification effect of the model. For example, for a model used in the industrial field to classify defect detection of components, if the recognition accuracy of the model is too low, the practicality of the model will be greatly reduced. Therefore, how to train the model in a semi-supervised learning mode or an unsupervised learning mode to improve the practicality of the model has always been a technical problem that technicians in this field need to solve. Summary of the invention

[0004] The present invention is proposed in view of the above problems. The present invention provides a model training method, a model training device, an electronic device, a storage medium and a computer program product.

[0005] According to one aspect of the present invention, a model training method is provided, the training method comprising:

[0006] For a first image in a first image set, a second image corresponding to the first image is determined in a second image set, and a first positive feature distance between a first feature corresponding to the first image and a second feature corresponding to the second image is determined, wherein the first image set and the second image set are image sets obtained by performing different data enhancement operations on a plurality of source images in the same image set, the source image of the first image and the source image of the second image are the same, and the first feature and the second feature are output by respectively inputting the first image and the second image into a model and being output by a feature extraction module of the model;

[0007] For a first image, a plurality of heterogeneous images are determined, and first feature distances between a first feature corresponding to the first image and a heterogeneous feature corresponding to each heterogeneous image are respectively determined, and the smallest first feature distance is used as a first negative feature distance corresponding to the first image, wherein a source image of the first image is different from a source image of the heterogeneous image, and the heterogeneous feature is obtained by inputting the heterogeneous image into a model and outputting it by a feature extraction module;

[0008] Determine a first loss value of the model according to a first positive example feature distance and a first negative example feature distance corresponding to each first image;

[0009] According to the first loss value, determine the final loss value of the model;

[0010] Use the final loss value to train the model.

[0011] Exemplarily, determining a first loss value of the model according to a first positive example feature distance and a first negative example feature distance corresponding to each first image includes:

[0012] Subtracting a first positive example feature distance corresponding to the first image from a first negative example feature distance corresponding to the first image to obtain a feature difference value corresponding to the first image;

[0013] The smallest feature difference value among the feature difference values ​​corresponding to all the first images is used as the first loss value.

[0014] Exemplarily, the multiple heterogeneous images include at least one of the following: a third image in a third image set and a second image in a second image set that is different from a source image of the first image, wherein each third image is labeled with a category label.

[0015] Exemplarily, before determining the final loss value of the model according to the first loss value, the method further includes:

[0016] For a third image in the third image set, a plurality of images of the same type are determined, and a second feature distance between a third feature corresponding to the third image and a third feature corresponding to each of the images of the same type is determined respectively, and the largest second feature distance is used as a second positive feature distance corresponding to the third image, wherein the annotated category label corresponding to the third image is the same as the category label corresponding to the images of the same type;

[0017] For the third image, a plurality of heterogeneous images are determined, and a third feature distance between a third feature corresponding to the third image and a third feature corresponding to each heterogeneous image is respectively determined, and the smallest third feature distance is used as a second negative feature distance corresponding to the third image, wherein the annotated category label corresponding to the third image is different from the category label corresponding to the heterogeneous image;

[0018] Determine a second loss value according to the second positive example feature distance and the second negative example feature distance corresponding to each third image;

[0019] According to the first loss value, the final loss value of the model is determined, including:

[0020] Determine a final loss value of the model according to the first loss value and the second loss value;

[0021] The third feature is obtained by respectively inputting the third image, the same type image and the different type image into the model and outputting them by the feature extraction module of the model.

[0022] Exemplarily, determining the second loss value according to the second positive example feature distance and the second negative example feature distance corresponding to each third image includes:

[0023] Subtract the second positive example feature distance corresponding to each third image from the second negative example feature distance corresponding to the third image to obtain a feature difference value corresponding to the third image;

[0024] The smallest feature difference value among the feature difference values ​​corresponding to all the third images is used as the second loss value.

[0025] Exemplarily, for a third image in a third image set, determining a plurality of images of the same type includes: determining other third images in the third image set that have the same annotated category label as the third image as at least a part of the plurality of images of the same type; and / or

[0026] For the third image, determining a plurality of heterogeneous images includes: determining third images in the third image set that have a different annotated category label from the third image as at least a part of the plurality of heterogeneous images.

[0027] Exemplarily, the above method further includes:

[0028] Performing a data augmentation operation on each third image in the third image set to obtain fourth images in the fourth image set, respectively, wherein the annotated category label corresponding to the third image is the same as the fourth image category label corresponding to the third image;

[0029] Among them, for the third image in the third image set, multiple similar images are determined, including: a fourth image with the same annotated category label as the third image is determined as at least a part of the multiple similar images; and / or for the third image, multiple different images are determined, including: a fourth image with a different annotated category label as the third image is determined as at least a part of the multiple different images.

[0030] Exemplarily, the above method further includes:

[0031] Training the model using a plurality of third images in the third image set to obtain an initially trained model, wherein each third image is labeled with a category label;

[0032] Input each first image in the first image set and each second image in the second image set into the initially trained model, respectively, so that the initially trained model outputs a first category prediction pseudo result for each first image and a second category prediction pseudo result for each second image;

[0033] Based on the first category prediction pseudo result and the second category prediction pseudo result, determining a first image with respective first category pseudo labels and a second image with respective second category pseudo labels;

[0034] Inputting first images with respective first category pseudo labels and second images with respective second category pseudo labels into the initially trained model respectively to obtain first category output results corresponding to each first image and second category output results corresponding to each second image;

[0035] Inputting a plurality of third images not used for initial training into the initially trained model respectively to obtain a third category output result corresponding to each third image not used for initial training;

[0036] Determine a class loss value of the initially trained model based on a difference between the first class pseudo label and the first class output result, a difference between the second class pseudo label and the second class output result, and a difference between the annotated class label and the third class output result;

[0037] According to the first loss value, the final loss value of the model is determined, including:

[0038] The final loss value of the model is determined based on the first loss value and the category loss value.

[0039] According to another aspect of the present invention, a model training device is also provided, which includes: a positive example feature distance determination module, a negative example feature distance determination module, a first loss value determination module, a final loss value determination module and a model training module.

[0040] The positive feature distance determination module is used for determining, for a first image in a first image set, a second image corresponding to the first image in a second image set, and determining a first positive feature distance between a first feature corresponding to the first image and a second feature corresponding to the second image, wherein the first image set and the second image set are image sets obtained by performing different data enhancement operations on a plurality of source images in the same image set, the source image of the first image and the source image of the second image are the same, and the first feature and the second feature are output by respectively inputting the first image and the second image into a model and being output by a feature extraction module of the model;

[0041] The negative example feature distance determination module is used to determine multiple heterogeneous images for the first image, respectively determine the first feature distance between the first feature corresponding to the first image and the heterogeneous features corresponding to each heterogeneous image, and use the smallest first feature distance as the first negative example feature distance corresponding to the first image, wherein the source image of the first image is different from the source image of the heterogeneous image, and the heterogeneous features are obtained by inputting the heterogeneous images into the model and outputting them by the feature extraction module;

[0042] The first loss value determination module is used to determine a first loss value of the model according to a first positive example feature distance and a first negative example feature distance corresponding to each first image;

[0043] The final loss value determination module is used to determine the final loss value of the model according to the first loss value;

[0044] The model training module is used to train the model using the final loss value.

[0045] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned model training method when the processor is running.

[0046] According to another aspect of an embodiment of the present invention, a storage medium is provided, storing a computer program / instruction, wherein the computer program / instruction is used to execute the above-mentioned model training method when running.

[0047] According to another aspect of an embodiment of the present invention, a computer program product is provided, including computer program instructions, which are used to execute the above-mentioned model training method when running.

[0048] According to the model training method, model training device, electronic device, storage medium and computer program product of the embodiment of the present invention, for the first image in the first image set, the second image corresponding to the first image can be determined in the second image set, and the first positive feature distance between the first feature corresponding to the first image and the second feature corresponding to the second image can be determined. Then, for the first image, multiple heterogeneous images are determined, and the feature distances between the first feature corresponding to the first image and the third feature corresponding to each heterogeneous image can be determined respectively, and the minimum feature distance is used as the first negative feature distance corresponding to the first image. Then, according to the first positive feature distance and the first negative feature distance corresponding to each first image, the first loss value can be determined. Then, according to the first loss value, the final loss value of the model can be determined, and then the model can be trained. The first positive feature distance in this scheme can be used to characterize the feature distance between different images obtained based on the same source image, and the first negative feature distance can be used to characterize the feature distance between different images obtained based on different source images. Therefore, training the model with the first loss value that can characterize the above feature distance can improve the classification accuracy of the model for the image to be tested, thereby improving the practicality of the model. In addition, the calculation of the first loss value does not depend on the label corresponding to the image, so it has strong versatility for both semi-supervised learning models and unsupervised learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0050] Figure 1 A schematic flow chart of a model training method according to an embodiment of the present invention is shown;

[0051] Figure 2 A schematic diagram showing a method for training a model according to an embodiment of the present invention is shown;

[0052] Figure 3 A schematic block diagram showing a training device for a model according to an embodiment of the present invention; and

[0053] Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0055] In the field of artificial intelligence, the accuracy of the classification results of a semi-supervised learning model (i.e., a machine learning model trained using a semi-supervised mode) is related to the accuracy of the pseudo-labels used in training. The accuracy of the classification results of an unsupervised learning model is related to the quality of the sample clustering used in training.

[0056] In industrial inspection scenarios, there are usually many types of defects on the objects to be tested, so there are also many types of pseudo-labels in the semi-supervised learning mode, and clustering is more difficult in the unsupervised learning mode. In the above scenarios, if there is less labeled data, the accuracy of the classification results output by the semi-supervised learning model and the unsupervised learning model will be difficult to meet practical standards.

[0057] In order to at least partially solve the above problems, an embodiment of the present invention provides a model training method. The above model can be a semi-supervised learning model or an unsupervised model that can perform classification tasks, such as an open world semi-supervised classification model, a Gaussian mixture model, etc. Figure 1 , Figure 1 FIG. 1 is a schematic flow chart of a method for training a model according to an embodiment of the present invention. Figure 1 As shown, the above training method may include steps S110 to S150.

[0058] In step S110, for a first image in the first image set, a second image corresponding to the first image is determined in the second image set, and a first positive feature distance between a first feature corresponding to the first image and a second feature corresponding to the second image is determined.

[0059] For example, for each first image u in the first image set i , determine in the second image set the same i The corresponding second image u i ', and determine the first image u i The corresponding first feature and second image u i 'The first positive feature distance between the corresponding second features. Where i is a positive integer less than or equal to the total number of source images, the first image u i The source image and the second image u i ' is the same as the source image.

[0060] The first image set and the second image set are image sets obtained by different data enhancement operations on multiple source images in the same image set. In one example, the source image may be an image of an object to be tested with defects. The object to be tested may be any type of item, such as a wafer (or a single crystal on a wafer), an electronic component, etc. The source image may be a static image or any video frame in a dynamic video. The source image may be an original image captured by an image acquisition device (such as an image sensor in a camera), or an image obtained after preprocessing the original image (such as digitization, normalization, smoothing, etc.). In some embodiments, if the object to be tested is a wafer, the defect category of the defect may include the presence of foreign matter in the wafer, cracks in the wafer, etc. The above-mentioned data enhancement operation may include: data enhancement operations of geometric transformation type (such as flipping, translation, jitter, etc.), data enhancement operations of pixel transformation type (such as color transformation, adding noise, blurring, etc.), etc.

[0061] In some embodiments, all first images in the first image set may use the same data augmentation operation, and all second images in the second data set may use the same data augmentation operation to ensure the consistency of the data sets, which helps reduce the probability of the model learning useless features.

[0062] The first feature and the second feature are respectively input into the model by the first image and the second image and output by the feature extraction module of the model. The feature extraction module is used to perform a feature extraction operation on the input first image or the second image to obtain the first feature or the second feature. In some embodiments, the architecture of the feature extraction module can be designed with reference to the convolution extraction architecture and the autoencoder extraction architecture, and the embodiments of the present invention are not described in detail here.

[0063] Since the first image u i The source image and the second image u i ' is the same as the source image, so the first image u i and the second image u i ' are regarded as positive examples of each other. The first positive example feature distance is used to represent the feature distance between the first image and the second image which are positive examples of each other. In the above example, the first image u i The corresponding first feature and second image u i 'The feature distance between the corresponding second features. By calculating the first positive example feature distance, the classification ability of the model for the positive example of the first image can be quantified. Optionally, the first positive example feature distance can be obtained by calculating the Euclidean distance or calculating the cosine similarity.

[0064] In step S120, for the first image, multiple heterogeneous images are determined, and first feature distances between the first feature corresponding to the first image and the heterogeneous features corresponding to each heterogeneous image are respectively determined, and the smallest first feature distance is used as the first negative example feature distance corresponding to the first image.

[0065] For example, for each first image u i , determine multiple heterogeneous images, and determine the first image u i The first feature distance between the corresponding first feature and the corresponding heterogeneous features of the heterogeneous image, and the smallest first feature distance is taken as the first image u i The corresponding first negative feature distance.

[0066] The first image u i The source image of the heterogeneous image is different from the source image of the heterogeneous image. In some embodiments, the heterogeneous image may include a first image set that is not the first image u i The heterogeneous features are obtained by inputting the heterogeneous images into the model and outputting them by the feature extraction module. The relevant description of the feature extraction module can be found above.

[0067] Since the first image u i The source image of the source image is different from the source image of the heterogeneous image, so the first image u i The first feature distance is used to represent the feature distance between the first image and the heterogeneous image, which are each other's negative examples. i The feature distance between the corresponding first feature and the corresponding heterogeneous features of the heterogeneous image. By calculating the first feature distance, the model can be quantified for the first image u i Optionally, the first feature distance can be obtained by calculating the Euclidean distance or the cosine similarity.

[0068] The first negative feature distance is used to indicate the image that is most easily recognized as the first image u i The positive example of the foreign source image (that is, the first image u i difficult sample) and the first image u i By calculating the first negative example feature distance, the model can be quantified for the first image u i The classification ability of difficult samples.

[0069] In step S130, a first loss value of the model is determined according to the first positive example feature distance and the first negative example feature distance corresponding to each first image.

[0070] The first positive feature distance is the distance between the first feature and the second feature obtained based on the same source image, so the training goal of the model may include reducing the first positive feature distance as much as possible so that the first image corresponding to the first feature and the second image corresponding to the second feature are regarded as the same image.

[0071] The first negative feature distance is the distance between the first feature and the heterogeneous feature obtained based on different source images, so the training goal of the model may include increasing the first negative feature distance as much as possible so that the first image corresponding to the first feature and the heterogeneous image corresponding to the heterogeneous feature are regarded as different images.

[0072] The first loss value can be determined based on the above training objectives. For example, the first positive feature distance corresponding to each first image is subtracted from the first negative feature distance corresponding to the first image to obtain the first difference value corresponding to each first image. Then the mean of the first difference values ​​corresponding to all first images is calculated as the first loss value. For another example, the mean of the first difference values ​​within a preset numerical range can be calculated as the first loss value to reduce the probability that abnormal values ​​outside the preset numerical range are noticed by the model.

[0073] In step S140, a final loss value of the model is determined according to the first loss value.

[0074] In one example, the first loss value may be directly used as the final loss value, or the first loss value may be combined with the loss values ​​calculated by other loss functions to obtain the above-mentioned final loss value.

[0075] In step S150, the model is trained using the final loss value.

[0076] The above-mentioned final loss value can be used to quantify the classification performance of the model. The smaller the final loss value, the better the classification performance of the model. The model parameters that need to be adjusted in the model can be updated by back propagation. After the model parameters are updated, if the model still needs to be trained, another batch of the first image set and the second image set can be used to perform the above steps S110 to S150 again to adjust the model parameters of the model again. After the model has been trained with several batches of the first image set and the second image set, if the user's needs are met, the training can be considered complete. The source images used between different batches of first image sets may be different, and the source images used between different batches of second image sets may be different.

[0077] According to the above scheme of the embodiment of the present invention, for each first image in the first image set, a second image can be determined in the second image set, and the first positive feature distance between the first feature corresponding to the first image and the second feature corresponding to the second image can be determined. Then, for each first image, multiple heterogeneous images are determined to respectively determine the feature distance between the first feature corresponding to the first image and the third feature corresponding to each heterogeneous image, and the minimum feature distance is used as the first negative feature distance corresponding to the first image. Then, according to the first positive feature distance and the first negative feature distance corresponding to each first image, a first loss value can be determined. Then, according to the first loss value, the final loss value of the model can be determined, and then the model can be trained. The first positive feature distance in the above scheme can be used to characterize the feature distance between different images obtained based on the same source image, and the first negative feature distance can be used to characterize the feature distance between different images obtained based on different source images. Therefore, training the model by using the first loss value that can characterize the above feature distance can improve the classification accuracy of the model for the image to be tested, thereby improving the practicality of the model. In addition, the calculation of the first loss value can be independent of the label corresponding to the image, so it has strong versatility for both semi-supervised learning models and unsupervised learning models.

[0078] Exemplarily, the plurality of heterogeneous images may include at least one of the following: a third image in the third image set and a second image in the second image set that is different from a source image of the first image.

[0079] For example, the plurality of heterogeneous images may include at least one of the following: a third image in the third image set and a second image in the second image set that is different from the source image of the first image. j '. Each third image may be labeled with a label category, j is a positive integer less than or equal to the total number of source images and j≠i. In an example, the third images in the third image set may be different from the images in the first image set and the second image set.

[0080] The annotated category label of the third image can be obtained by manual labeling or labeling by other machine learning models. The above category label can be used to indicate the category to which the third image belongs. For example, the category label can be used to indicate the defect category mentioned above. The third image in the third image set is independent of the source image used to obtain the first image, so the third image can also be used as one of the heterogeneous images to increase the number of heterogeneous images.

[0081] According to the above solution of the embodiment of the present invention, the number of heterogeneous images corresponding to the first image can be increased by adding the second image and / or the third image to the heterogeneous image. Therefore, the model trained based on the above heterogeneous images has a strong classification performance when facing images under different transformations, and can improve the generalization ability of the model when classifying unseen images.

[0082] Exemplarily, step S130 determines the first loss value of the model according to the first positive example feature distance and the first negative example feature distance corresponding to each first image, and may include step S131 and step S132.

[0083] In step S131, a first positive example feature distance corresponding to the first image is subtracted from a first negative example feature distance corresponding to the first image to obtain a feature difference value corresponding to the first image.

[0084] For example, each first image u i The corresponding first positive feature distance is the first image u i The corresponding first negative feature distance is subtracted to obtain the first image u i The corresponding feature difference.

[0085] In one example, the first positive example feature distance and the first negative example feature distance may be feature distances of the same type, so as to process data in the same dimension, which is conducive to improving the interpretability of feature differences. For example, the first positive example feature distance and the first negative example feature distance may both be Euclidean distances or cosine similarities, and the two may be directly subtracted to obtain the above feature difference.

[0086] In step S132, the smallest feature difference value among the feature difference values ​​corresponding to all the first images is used as the first loss value.

[0087] The above-mentioned minimum feature difference is used to represent the feature difference between the closest first positive example feature distance and the first negative example feature distance, and is the most representative feature difference. The minimum feature difference is used as the first loss value, that is, the first loss value is positively correlated with the first positive example feature distance and negatively correlated with the first negative example feature distance. Combined with the training scenario of the model, the model can reduce the first loss value through iteration, so it is equivalent to the model reducing the first positive example feature distance and increasing the first negative example feature distance after iteration. That is, after iteration, the model will try its best to regard the first image and the second image obtained based on the same source image as the same image, and the first image and the heterogeneous image obtained based on different source images as different images.

[0088] According to the above scheme of the embodiment of the present invention, the first positive example feature distance and the first negative example feature distance can be subtracted to obtain a feature difference. Then the smallest feature difference value among all feature differences is used as the first loss value to train the model. The positive and negative examples of the first image corresponding to the smallest feature difference value are the most difficult to distinguish, so they are more representative. Using the smallest feature difference value as the first loss value to train the model is conducive to improving the classification accuracy of the model for the above difficult-to-distinguish images.

[0089] Exemplarily, before determining the final loss value of the model according to the first loss value in step S140, the training method provided by the embodiment of the present invention may further include steps S210 to S230.

[0090] In step S210, for the third image in the third image set, multiple similar images are determined, and the second feature distances between the third feature corresponding to the third image and the third feature corresponding to each similar image are respectively determined, and the largest second feature distance is used as the second positive example feature distance corresponding to the third image.

[0091] For example, for each third image X in the third image set k , determine multiple similar images, and determine the third image X respectively k The second feature distance between the corresponding third feature and the third feature corresponding to each similar image, and the largest second feature distance is taken as the third image X k The corresponding second positive feature distance. Wherein k is a positive integer less than or equal to the total number of the third image. The third image X k The corresponding annotated category label is the same as the category label corresponding to the same type of image. The data source of the same type of image is not limited here, for example, it can be an image in another image set (different from the first image set, the second image set, and the third image set). Since the third image has the same category label as the same type of image, the third image and the same type of image can be regarded as positive examples of each other.

[0092] The third features of the third image and the image of the same type can be respectively obtained by inputting the third image and the image of the same type into the model and then using the feature extraction module of the model.

[0093] The second characteristic distance is used to represent the characteristic distance between the third image and the same type of image which are positive examples of each other. In the above example, the third image X k The characteristic distance between the corresponding third feature and the third features corresponding to each of the plurality of similar images. By calculating the second characteristic distance, the classification ability of the model for the positive example of the third image can be quantified. Optionally, the second characteristic distance can be obtained by calculating the Euclidean distance or the cosine similarity.

[0094] The second positive feature distance is used to indicate the image that is most easily recognized as the third image X k The negative example of the same image (that is, the third image X k One of the difficult samples) and the third image X k By calculating the feature distance of the second positive example, the model can be quantified for the third image X k The classification ability of difficult samples.

[0095] In step S220, for the third image, multiple heterogeneous images are determined, and the third feature distances between the third feature corresponding to the third image and the third feature corresponding to each heterogeneous image are respectively determined, and the minimum third feature distance is used as the second negative example feature distance corresponding to the third image.

[0096] For example, for each third image X in the third image set k , determine multiple heterogeneous images, and determine the third image X respectively k The third feature distance between the corresponding third feature and the third feature corresponding to each heterogeneous image, and the smallest third feature distance is taken as the third image X k The corresponding second negative example feature distance.

[0097] Third Image X k The corresponding annotated category label is different from the category label corresponding to the heterogeneous image. Since the category labels of the third image and the heterogeneous image are different, the third image and the heterogeneous image can be regarded as negative examples of each other.

[0098] The third feature of the heterogeneous image can be obtained by inputting the heterogeneous image into the model and outputting it by the feature extraction module of the model.

[0099] The third characteristic distance is used to represent the characteristic distance between the third image and the heterogeneous image, which are negative examples of each other. In the above example, the third image X k The feature distance between the corresponding third feature and the third features corresponding to each of the plurality of heterogeneous images. By calculating the third feature distance, the classification ability of the model for the negative example of the third image can be quantified. Optionally, the third feature distance can be obtained by calculating the Euclidean distance or the cosine similarity.

[0100] The second negative feature distance is used to indicate the image that is most easily identified as the third image X k The positive heterogeneous image (that is, the third image X k One of the difficult samples) and the third image X k By calculating the feature distance of the second negative example, the model can be quantified for the third image X k The classification ability of difficult samples.

[0101] In step S230, a second loss value is determined according to the second positive example feature distance and the second negative example feature distance corresponding to each third image.

[0102] The second positive example feature distance is the feature distance between the third features belonging to the same category label, so the training goal of the model may include reducing the second positive example feature distance as much as possible to regard the third image and the same type of image as images of the same category.

[0103] The second negative example feature distance is the feature distance between the third features belonging to different category labels, so the training goal of the model may include increasing the second negative example feature distance as much as possible to regard the third image and the heterogeneous image as images of different categories.

[0104] The second loss value can be determined based on the above training objectives. For example, the second positive feature distance corresponding to each third image is subtracted from the second negative feature distance corresponding to the third image to obtain the second difference value corresponding to each third image. Then the mean of the second difference values ​​corresponding to all third images is calculated as the second loss value. For another example, the mean of the second difference values ​​within a preset numerical range can be calculated as the second loss value to reduce the probability that abnormal values ​​outside the preset numerical range are learned by the model.

[0105] If the training method includes steps S210 to S230, step S140 determines the final loss value of the model according to the first loss value, which may include: determining the final loss value of the model according to the first loss value and the second loss value.

[0106] The final loss value may be positively correlated with the first loss value and the second loss value, respectively. For example, the final loss value may be a value obtained by summing or weighted summing the first loss value and the category loss value.

[0107] According to the above scheme of the embodiment of the present invention, the second positive feature distance between the third image and the same type of image and the second negative feature distance between the third image and the different type of image can be determined, thereby obtaining a second loss value. The final loss value of the model is then determined based on the first loss value and the second loss value. The above scheme uses the third image with annotated category labels in the training process of the model, which can improve the model's classification ability for images with different category labels, and is conducive to improving the classification accuracy of the model.

[0108] Exemplarily, determining a plurality of similar images for the third image in the third image set in step S210 may include: determining other third images in the third image set that have the same annotated category label as the third image as at least a part of the plurality of similar images.

[0109] Combined with the relevant content of step S210 above, the third image X k For example, the above-mentioned multiple similar images may include: k The corresponding third image with the same annotated category label and the third image X in other datasets k Corresponding images with the same annotated category labels.

[0110] According to the above scheme of the embodiment of the present invention, the third image in the third image set can be used as at least a part of a plurality of similar images to increase the number of similar images, which is conducive to improving the representativeness of the second characteristic distance generated based on similar images. Since the shooting environments of the images in the same data set are similar (for example, similar lighting, background, etc.), the above scheme is conducive to improving the classification performance of the model under the shooting environment of the third data set. Especially in the scenario where the defect category of the wafer is detected by the model, the shooting environment of the wafer image usually does not change (for example, the shooting angle of the image acquisition device for shooting the wafer image remains unchanged, the wafer carrier in the image background remains unchanged, and the lighting conditions are made uniform by the fill light device, etc.), so the above model is more adaptable to the actual shooting environment.

[0111] Exemplarily, the training method provided by the embodiment of the present invention may further include: performing a data enhancement operation on each third image in the third image set, respectively, to obtain fourth images corresponding to the third images in the fourth image set.

[0112] For different third images, the same data augmentation operation can be used to ensure data consistency. The annotated category label corresponding to the third image is the same as the category label of the fourth image corresponding to the third image.

[0113] If the fourth data set has been obtained, determining a plurality of similar images for the third image in the third image set in step S210 may include: determining a fourth image having the same annotated category label as the third image as at least a part of the plurality of similar images.

[0114] Take the third image X k For example, in combination with the relevant content of step S210 above, the above-mentioned multiple similar images may include: k The corresponding third images with the same annotated category label, the third image X in the fourth data set k The fourth image with the same corresponding annotated category label, and the third image X in other data sets (different from the first image set, the second image set, the third image set, and the fourth image set in this example) k Corresponding images with the same annotated category labels.

[0115] According to the above scheme of the embodiment of the present invention, the fourth image in the fourth image set can be used as at least a part of multiple images of the same type to increase the number of images of the same type, which is conducive to improving the representativeness of the second characteristic distance generated based on the images of the same type. In addition, the fourth image is obtained based on the third image through data enhancement operation, and the shooting environment is the same as the shooting environment of the third image. Therefore, on the one hand, it is conducive to improving the classification performance of the model in the shooting environment (which can be combined with the shooting environment of the wafer image mentioned above); on the other hand, it is also conducive to improving the generalization ability of the model in the shooting environment.

[0116] Exemplarily, determining a plurality of heterogeneous images for the third image in step S220 may include: determining a third image in the third image set that has a different annotated category label from the third image as at least a part of the plurality of heterogeneous images.

[0117] Combined with the relevant content of step S220 above, the third image X k For example, the plurality of heterogeneous images may include: k The corresponding third image with different annotated category labels and the third image X in other data sets k The corresponding images have different annotated category labels.

[0118] According to the above scheme of the embodiment of the present invention, the third image in the third data set can be used as at least a part of the multiple heterogeneous images to increase the number of heterogeneous images, which is conducive to improving the representativeness of the third feature distance generated based on the heterogeneous images. Since the shooting environments of the images in the same data set are similar, the above scheme is conducive to improving the classification performance of the model in the shooting environment of the third data set. Especially in the scenario where the defect category of the wafer is detected by the model, the shooting environment of the wafer image usually does not change, so the above model is more adaptable to the actual shooting environment.

[0119] Exemplarily, if the fourth image set mentioned above has been obtained, determining multiple heterogeneous images for the third image in step S220 may include: determining a fourth image having a different annotated category label from the third image as at least a part of the multiple heterogeneous images.

[0120] Take the third image X k For example, in combination with the relevant content of step S220 above, the above-mentioned multiple heterogeneous images may include: k The corresponding third image with different annotated category labels, the fourth image set, and the third image X kThe fourth image with a different corresponding labeled category label, and the third image X in other data sets (different from the first image set, the second image set, the third image set, and the fourth image set in this example) k The corresponding images have different annotated category labels.

[0121] According to the above scheme of the embodiment of the present invention, the fourth image in the fourth image set can be used as at least a part of the multiple heterogeneous images to increase the number of heterogeneous images, which is conducive to improving the representativeness of the third characteristic distance generated based on the heterogeneous images. In addition, the fourth image is obtained based on the third image through data enhancement operation, and the shooting environment is the same as the shooting environment of the third image. Therefore, on the one hand, it is conducive to improving the classification performance of the model in this shooting environment (which can be combined with the shooting environment of the wafer image mentioned above); on the other hand, it is also conducive to improving the generalization ability of the model in this shooting environment.

[0122] Exemplarily, step S230 determines the second loss value according to the second positive example feature distance and the second negative example feature distance corresponding to each third image, and may include steps S231 and S232.

[0123] In step S231, the second positive example feature distance corresponding to each third image is subtracted from the second negative example feature distance corresponding to the third image to obtain a feature difference value corresponding to the third image.

[0124] For example, each third image X k The corresponding second positive feature distance and the third image X k The corresponding second negative feature distance is subtracted to obtain the third image X k The corresponding feature difference.

[0125] In one example, the second positive example feature distance and the second negative example feature distance may be feature distances of the same type, so as to process data in the same dimension, which is conducive to improving the interpretability of the feature difference. For example, the second positive example feature distance and the second negative example feature distance may both be Euclidean distances or cosine similarities, and the two may be directly subtracted to obtain the above feature difference.

[0126] In step S232, the smallest feature difference value among the feature difference values ​​corresponding to all the third images is used as the second loss value.

[0127] The above-mentioned minimum feature difference is used to represent the feature difference between the closest second positive example feature distance and the second negative example feature distance, and is the most representative feature difference. The minimum feature difference is used as the second loss value, that is, the second loss value is positively correlated with the second positive example feature distance and negatively correlated with the second negative example feature distance. Combined with the training scenario of the model, the model can reduce the second loss value through iteration, so it is equivalent to the model reducing the second positive example feature distance and increasing the second negative example feature distance after iteration. That is, after iteration, the model will try its best to regard the third image and similar images as images belonging to the same category label, and regard the third image and heterogeneous images as images belonging to different category labels.

[0128] According to the above scheme of the embodiment of the present invention, the second positive example feature distance and the second negative example feature distance can be subtracted to obtain a feature difference. Then, the smallest feature difference value among all feature differences is used as the second loss value to train the model. The positive and negative examples of the third image corresponding to the smallest feature difference value are the most difficult to distinguish, so they are more representative. Using the smallest feature difference value as the second loss value to train the model is conducive to improving the classification accuracy of the model for the above difficult-to-distinguish images.

[0129] Exemplarily, the training method provided by the embodiment of the present invention may further include: steps S310 to S360.

[0130] In step S310, the model is trained using a plurality of third images in the third image set to obtain an initially trained model.

[0131] The plurality of third images may be images in the third image set described above, and each third image is labeled with a category label. In one example, the plurality of third images in the third image set may be divided into a plurality of batches, and some of the third images in the plurality of batches may be used to perform initial training on the model, and other batches of third images may be used to generate the third category output result in step S350.

[0132] The embodiment of the present invention provides a process of initial training of a model for reference: the third images in the same batch are input into the model to obtain the predicted label corresponding to each third image in the batch. The predicted label corresponding to each third image is compared with the annotated category label corresponding to the third image to obtain the loss value to adjust the model parameters of the model. Repeat the above steps using another batch of third images. After training with several batches of third images, if the model meets the user requirements, the model can be used as the initially trained model.

[0133] In step S320, each first image in the first image set and each second image in the second image set are respectively input into the initially trained model, so that the initially trained model outputs a first category prediction pseudo result for each first image and a second category prediction pseudo result for each second image.

[0134] Figure 2 FIG. 1 is a schematic diagram showing a method for training a model according to an embodiment of the present invention. Figure 2 , the source image (see Figure 2 The first image in the first image set and the second image in the second image set are obtained by different data augmentation methods. Then, the model can output the first category prediction pseudo result and the second category prediction pseudo result.

[0135] The first category prediction pseudo result and the second category prediction pseudo result may include the confidence corresponding to each category label, respectively. In one example, the model may include two classification modules, wherein one classification module A is used to generate the first category prediction pseudo result for the first image. Another classification module B is used to generate the second category prediction pseudo result for the second image.

[0136] In step S330 , based on the first category prediction pseudo result and the second category prediction pseudo result, first images with respective first category pseudo labels and second images with respective second category pseudo labels are determined.

[0137] In some embodiments, the category label with the highest confidence in the first category prediction pseudo result can be used as the first category pseudo label of the first image corresponding to the first category prediction pseudo result. The category label with the highest confidence in the second category prediction pseudo result can be used as the second category pseudo label of the second image corresponding to the second category prediction pseudo result. Alternatively, the category label with the highest confidence in the first category prediction pseudo result and greater than a preset value can be used as the first category pseudo label of the first image corresponding to the first category prediction pseudo result. The category label with the highest confidence in the second category prediction pseudo result and greater than a preset value can be used as the second category pseudo label of the second image corresponding to the second category prediction pseudo result. The above preset value can be set by the developer. The higher the preset value, the fewer first images with first category pseudo labels and second images with second category pseudo labels will be, but the representativeness of the above images will be higher. Figure 2 In the example of , the confidences in the first category prediction pseudo results and the second category prediction pseudo results can be processed by the Sinkhorn-Knopp algorithm to generate the first category pseudo labels and the second category pseudo labels.

[0138] In step S340, the first images with respective first category pseudo labels (see Figure 2The first set of images with pseudo labels in the figure) and the second set of images with their respective second category pseudo labels (see Figure 2 A second image set with pseudo labels in the first image) is respectively input into the initially trained model to obtain a first category output result corresponding to each first image and a second category output result corresponding to each second image.

[0139] The above-mentioned first category output result and the above-mentioned second category output result can both be category labels. In some embodiments, if the model includes multiple classification modules, the first category output result and the second category output result can be generated by different classification modules respectively. Furthermore, the classification module for generating the first category pseudo-label can be used to generate the second category output result, and the classification module for generating the second category pseudo-label can be used to generate the first category output result, so that collaborative training between classification modules can be achieved. In conjunction with the example of the above model including classification module A and classification module B, the first image can be input into classification module B to obtain the first category output result corresponding to each first image, and the second image can be input into classification module A to obtain the second category output result corresponding to each second image.

[0140] In step S350, a plurality of third images (see Figure 2 The third image set with labels in the initial training) are respectively input into the initially trained model to obtain the third category output results corresponding to each third image not used for initial training.

[0141] The third category output result may be a category label. In some embodiments, if the model includes multiple classification modules, the third category output result of the third image may be a comprehensive result obtained based on the classification results of each of the multiple classification modules. In combination with the example in which the model includes classification module A and classification module B, classification module A and classification module B may generate category label a and category label b based on the third image, respectively. If category label a and category label b are the same category label, the third category output result is the category label; if category label a and category label b are different category labels, the third category output result may be further determined based on their respective confidences. Alternatively, classification module A and classification module B may generate confidence distribution a and confidence distribution b (confidence distribution includes the confidence corresponding to each category label) based on the third image, respectively. For a category label, the confidence a and confidence b corresponding to the category label are determined in confidence distribution a and confidence distribution b, respectively. The mean of confidence a and confidence b is calculated (or the weighted mean is calculated), and then the category label with the highest mean can be output as the third category.

[0142] In step S360, the category loss value of the initially trained model is determined based on the difference between the first category pseudo label and the first category output result, the difference between the second category pseudo label and the second category output result, and the difference between the marked category label and the third category output result.

[0143] The above-mentioned category loss value may be positively correlated with the difference between the first category pseudo label and the first category output result, the difference between the second category pseudo label and the second category output result, and the difference between the labeled category label and the third category output result. In some embodiments, the above-mentioned category loss value may be calculated by a cross-entropy loss function (Cross-Entropy Loss), or by an improved cross-entropy loss function (e.g., Focal Loss).

[0144] If the training method includes steps S310 to S360, step S140 determines the final loss value of the model according to the first loss value, which may include: determining the final loss value of the model according to the first loss value and the category loss value.

[0145] The final loss value may be positively correlated with the first loss value and the category loss value, respectively. For example, the final loss value may be a value obtained by summing or weighted summing the first loss value and the category loss value. In some embodiments, if the training method also includes steps S210 to S230, the final loss value may also be positively correlated with the first loss value, the second loss value, and the category loss value, respectively. In other embodiments, the final loss value includes a consistency loss value, and the consistency loss value includes the first loss value and the second loss value mentioned above. In some further embodiments, the fourth loss value may also be calculated based on the third image set and the fourth image set in a manner similar to the first image set and the second image set. The above-mentioned fourth loss value is used to indicate whether the model will recognize the third image and the fourth image obtained through data augmentation operations based on the third image as the same image. The generation method of the first loss value mentioned above may be referred to, and the embodiments of the present invention will not be elaborated here. In this example, the consistency loss value may be obtained based on the first feature, the second feature, and the fourth feature (the image feature of the third image). In Figure 2 In the example, the model can be updated based on at least EMA (exponential moving average).

[0146] According to the above scheme provided by the embodiment of the present invention, the model can be initially trained by a third image with annotated category labels. Then, based on the first image and the second image, a first image with a first category pseudo label and a second image with a second category pseudo label are generated, and the category loss value can be obtained. Then, based on the first loss value and the category loss value, the final loss value of the model is determined. On the one hand, by generating the first category pseudo label and the second category pseudo label, the number of images used by the model in the training stage can be increased, which is beneficial to improving the generalization ability of the trained model; on the other hand, training the model based on the above final loss value can improve the model's classification ability for images with different category labels, which is beneficial to improving the classification accuracy of the model.

[0147] According to another aspect of the present invention, a model training device is also provided. Figure 3 , Figure 3 A schematic block diagram of a model training device according to an embodiment of the present invention is shown. The training device 400 includes: a positive example feature distance determination module 410, a negative example feature distance determination module 420, a first loss value determination module 430, a final loss value determination module 440 and a model training module 450.

[0148] The positive feature distance determination module 410 is used to determine the second image corresponding to the first image in the second image set for the first image in the first image set, and determine the first positive feature distance between the first feature corresponding to the first image and the second feature corresponding to the second image. The first image set and the second image set are respectively image sets obtained by different data enhancement operations on multiple source images in the same image set, the source image of the first image and the source image of the second image are the same, and the first feature and the second feature are respectively input into the model and output by the feature extraction module of the model. The negative feature distance determination module 420 is used to determine multiple heterogeneous images for the first image, respectively determine the first feature distance between the first feature corresponding to the first image and the heterogeneous feature corresponding to each heterogeneous image, and take the minimum first feature distance as the first negative feature distance corresponding to the first image. The source image of the first image is different from the source image of the heterogeneous image, and the heterogeneous feature is input into the model and output by the feature extraction module. The first loss value determination module 430 is used to determine the first loss value of the model according to the first positive feature distance and the first negative feature distance corresponding to each first image. The final loss value determination module 440 is used to determine the final loss value of the model according to the first loss value. The model training module 450 is used to train the model using the final loss value.

[0149] Exemplarily, the first loss value determining module 430 may include: a first feature difference value generating module and a first loss value generating module.

[0150] The first feature difference value generation module is used to subtract the first positive feature distance corresponding to the first image from the first negative feature distance corresponding to the first image to obtain the feature difference value corresponding to the first image. The first loss value generation module is used to take the smallest feature difference value among the feature differences corresponding to all the first images as the first loss value.

[0151] Exemplarily, the multiple heterogeneous images include at least one of the following: a third image in the third image set and a second image in the second image set that is different from the source image of the first image, wherein each third image is labeled with a category label.

[0152] Exemplarily, the training device 400 further includes: a second positive example feature distance determination module, a second negative example feature distance determination module and a second loss value determination module.

[0153] The second positive example feature distance determination module is used to determine multiple similar images for the third image in the third image set, and respectively determine the second feature distance between the third feature corresponding to the third image and the third feature corresponding to each similar image, and use the largest second feature distance as the second positive example feature distance corresponding to the third image. The annotated category label corresponding to the third image is the same as the category label corresponding to the similar image. The second negative example feature distance determination module is used to determine multiple heterogeneous images for the third image, and respectively determine the third feature distance between the third feature corresponding to the third image and the third feature corresponding to each heterogeneous image, and use the smallest third feature distance as the second negative example feature distance corresponding to the third image. The annotated category label corresponding to the third image is different from the category label corresponding to the heterogeneous image. The second loss value determination module is used to determine the second loss value according to the second positive example feature distance and the second negative example feature distance corresponding to each third image. The final loss value determination module 440 is also used to determine the final loss value of the model according to the first loss value and the second loss value. The third feature is respectively input into the model by the third image, the similar image and the heterogeneous image and output by the feature extraction module of the model.

[0154] Exemplarily, the second loss value determining module may include: a second feature difference generating module and a second loss value first submodule.

[0155] The second feature difference value generation module is used to subtract the second positive feature distance corresponding to each third image from the second negative feature distance corresponding to the third image to obtain the feature difference value corresponding to the third image. The second loss value first submodule is used to take the smallest feature difference value among the feature differences corresponding to all the third images as the second loss value.

[0156] Exemplarily, the second positive example feature distance determination module may include: a first similar image determination module, and / or a first heterogeneous image determination module. The first similar image determination module is used to determine other third images in the third image set that have the same annotated category label as the third image as at least a portion of the multiple similar images. The first heterogeneous image determination module is used to determine third images in the third image set that have a different annotated category label than the third image as at least a portion of the multiple heterogeneous images.

[0157] Exemplarily, the training device 400 may further include: a fourth image generating module.

[0158] The fourth image generation module is used to perform data enhancement operations on each third image in the third image set to obtain fourth images in the fourth image set, wherein the annotated category label corresponding to the third image is the same as the category label corresponding to the fourth image corresponding to the third image.

[0159] The second positive example feature distance determination module may include: a second same type image determination module and / or a second different type image determination module.

[0160] The second same-class image determination module is used to determine a fourth image with the same annotated category label as the third image as at least a part of the plurality of same-class images. The second different-class image determination module is used to determine a fourth image with a different annotated category label as the third image as at least a part of the plurality of different-class images.

[0161] Exemplarily, the training device 400 further includes: an initial training module, a category prediction pseudo result generation module, a category pseudo label determination module, a first and second category output result generation module, a third category output result generation module and a category loss value determination module.

[0162] The initial training module is used to train the model using a plurality of third images in the third image set to obtain an initially trained model, wherein each third image is labeled with a category label.

[0163] The category prediction pseudo-result generation module is used to input each first image in the first image set and each second image in the second image set into the initially trained model, respectively, so that the initially trained model outputs the first category prediction pseudo-result of each first image and the second category prediction pseudo-result of each second image.

[0164] The category pseudo-label determination module is used to determine first images with respective first category pseudo-labels and second images with respective second category pseudo-labels based on the first category prediction pseudo-results and the second category prediction pseudo-results.

[0165] The first and second category output result generation module is used to input the first images with respective first category pseudo labels and the second images with respective second category pseudo labels into the initially trained model respectively to obtain the first category output result corresponding to each first image and the second category output result corresponding to each second image.

[0166] The third category output result generating module is used to input a plurality of third images not used for initial training into the initially trained model respectively, so as to obtain a third category output result corresponding to each third image not used for initial training.

[0167] The category loss value determination module is used to determine the category loss value of the initially trained model based on the difference between the first category pseudo label and the first category output result, the difference between the second category pseudo label and the second category output result, and the difference between the marked category label and the third category output result.

[0168] The final loss value determination module 440 is further used to determine the final loss value of the model according to the first loss value and the category loss value.

[0169] According to yet another aspect of the present invention, an electronic device is provided. Figure 4 FIG. 1 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Figure 4 As shown, the electronic device 500 includes a processor 510 and a memory 520, in which a computer program is stored. When the computer program instructions are executed by the processor 510, the training method is executed.

[0170] According to another aspect of the present invention, a storage medium is provided, storing a computer program / instruction, and the storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, an erasable programmable read-only memory (EPROM), a portable read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The storage medium may be any combination of one or more computer-readable storage media. The computer program / instruction is used by a processor to execute the above training method when it is running.

[0171] A person skilled in the art may understand the specific implementation scheme of the above-mentioned electronic device and storage medium by reading the above-mentioned description of the training method, which will not be described here for the sake of brevity.

[0172] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0173] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0174] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0175] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0176] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0177] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0178] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0179] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in a training device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0180] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0181] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: The method comprises: For a first image in a first image set, a second image corresponding to the first image is determined in a second image set, and a first positive feature distance between a first feature corresponding to the first image and a second feature corresponding to the second image is determined, wherein the first image set and the second image set are image sets obtained by performing different data enhancement operations on a plurality of source images in the same image set, the source image of the first image is the same as the source image of the second image, and the first feature and the second feature are output by a feature extraction module of the model by respectively inputting the first image and the second image into the model; For the first image, a plurality of heterogeneous images are determined, and first feature distances between a first feature corresponding to the first image and a heterogeneous feature corresponding to each heterogeneous image are respectively determined, and the smallest first feature distance is used as a first negative feature distance corresponding to the first image, wherein a source image of the first image is different from a source image of the heterogeneous image, and the heterogeneous feature is obtained by inputting the heterogeneous image into the model and outputted by the feature extraction module; Determining a first loss value of the model according to a first positive example feature distance and a first negative example feature distance corresponding to each first image; Determining a final loss value of the model according to the first loss value; The model is trained using the final loss value.

2. The method according to claim 1, characterized in that The determining the first loss value of the model according to the first positive example feature distance and the first negative example feature distance corresponding to each first image includes: Subtracting a first positive example feature distance corresponding to the first image from a first negative example feature distance corresponding to the first image to obtain a feature difference value corresponding to the first image; The smallest feature difference value among the feature difference values ​​corresponding to all the first images is used as the first loss value.

3. The method according to claim 1, characterized in that The multiple heterogeneous images include at least one of the following: a third image in a third image set and a second image in the second image set that is different from a source image of the first image, wherein each third image is labeled with a category label.

4. The method according to claim 1, characterized in that Before determining the final loss value of the model according to the first loss value, the method further includes: For a third image in the third image set, a plurality of images of the same type are determined, and a second feature distance between a third feature corresponding to the third image and a third feature corresponding to each image of the same type is respectively determined, and the largest second feature distance is used as a second positive feature distance corresponding to the third image, wherein the annotated category label corresponding to the third image is the same as the category label corresponding to the image of the same type; For the third image, determine a plurality of heterogeneous images, and respectively determine a third feature distance between a third feature corresponding to the third image and a third feature corresponding to each heterogeneous image, and use the smallest third feature distance as a second negative feature distance corresponding to the third image, wherein the annotated category label corresponding to the third image is different from the category label corresponding to the heterogeneous image; Determine a second loss value according to the second positive example feature distance and the second negative example feature distance corresponding to each third image; Determining a final loss value of the model according to the first loss value includes: Determining a final loss value of the model according to the first loss value and the second loss value; The third feature is obtained by respectively inputting the third image, the same type image and the different type image into the model and outputting them by the feature extraction module of the model.

5. The method according to claim 4, characterized in that The determining the second loss value according to the second positive example feature distance and the second negative example feature distance corresponding to each third image includes: Subtract the second positive example feature distance corresponding to each third image from the second negative example feature distance corresponding to the third image to obtain a feature difference value corresponding to the third image; The smallest feature difference value among the feature difference values ​​corresponding to all the third images is used as the second loss value.

6. The method according to claim 4, characterized in that The determining of a plurality of images of the same type for the third image in the third image set includes: determining other third images in the third image set that have the same annotated category label as that of the third image as at least a part of the plurality of images of the same type; and / or The determining of a plurality of heterogeneous images for the third image includes: determining third images in the third image set that have a different annotated category label from that corresponding to the third image as at least a part of the plurality of heterogeneous images.

7. The method according to any one of claims 4 to 6, characterized in that The method further comprises: Performing data enhancement operations on each third image in the third image set to obtain fourth images corresponding to the third image in the fourth image set, wherein the annotated category label corresponding to the third image is the same as the category label of the fourth image corresponding to the third image; The step of determining a plurality of similar images for the third image in the third image set includes: determining a fourth image having the same annotated category label as that corresponding to the third image as at least a part of the plurality of similar images; and / or the step of determining a plurality of different images for the third image includes: determining a fourth image having a different annotated category label as that corresponding to the third image as at least a part of the plurality of different images.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Training the model using a plurality of third images in a third image set to obtain an initially trained model, wherein each third image is labeled with a category label; Input each first image in the first image set and each second image in the second image set into the initially trained model, respectively, so that the initially trained model outputs a first category prediction pseudo result for each first image and a second category prediction pseudo result for each second image; Based on the first category prediction pseudo result and the second category prediction pseudo result, determining a first image with respective first category pseudo labels and a second image with respective second category pseudo labels; Inputting the first images with respective first category pseudo labels and the second images with respective second category pseudo labels into the initially trained model respectively, so as to obtain the first category output result corresponding to each first image and the second category output result corresponding to each second image; Inputting a plurality of third images not used for initial training into the initially trained model respectively to obtain a third category output result corresponding to each third image not used for initial training; Determining a category loss value of the initially trained model according to a difference between the first category pseudo label and the first category output result, a difference between the second category pseudo label and the second category output result, and a difference between the marked category label and the third category output result; Determining a final loss value of the model according to the first loss value includes: A final loss value of the model is determined based on the first loss value and the category loss value.

9. A model training device, characterized in that: The device comprises: a positive feature distance determination module, for determining, for a first image in a first image set, a second image corresponding to the first image in a second image set, and determining a first positive feature distance between a first feature corresponding to the first image and a second feature corresponding to the second image, wherein the first image set and the second image set are image sets obtained by performing different data enhancement operations on a plurality of source images in the same image set, the source image of the first image and the source image of the second image are the same, and the first feature and the second feature are output by a feature extraction module of the model by respectively inputting the first image and the second image into the model; a negative example feature distance determination module, configured to determine, for the first image, a plurality of heterogeneous images, respectively determine a first feature distance between a first feature corresponding to the first image and a heterogeneous feature corresponding to each heterogeneous image, and use the smallest first feature distance as a first negative example feature distance corresponding to the first image, wherein a source image of the first image is different from a source image of the heterogeneous image, and the heterogeneous feature is obtained by inputting the heterogeneous image into the model and outputted by the feature extraction module; A first loss value determination module, used to determine a first loss value of the model according to a first positive example feature distance and a first negative example feature distance corresponding to each first image; A final loss value determination module, used to determine a final loss value of the model according to the first loss value; The model training module is used to train the model using the final loss value.

10. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the model training method as described in any one of claims 1 to 8 when executed.

11. A storage medium storing a computer program / instruction, characterized in that: The computer program / instructions are used to execute the training method of the model as described in any one of claims 1 to 8 when running.

12. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the training method of the model as described in any one of claims 1 to 8 when running.

Images