Image processing model training method, image processing method, and electronic device

Abstract

The application discloses an image processing model training method, an image processing method and an electronic device, relates to the technical field of image processing model training, and comprises the following steps: obtaining a current batch of samples in a training image set, removing the category labels of the samples to obtain a first category set, and extracting a first category center vector set from a target category center matrix; a second category set is formed by the remaining category labels in the training image set which do not belong to the first category set, a category is sampled from the second category set according to an importance weight, a second category center vector set is extracted, and the importance weight is determined according to the correlation degree between a category center vector and a sample feature vector and the gradient norm of the category center vector in a preset number of iterations; the first and second category center vector sets are merged, and the training parameters are updated according to the loss value calculated by the sample feature vector and the merged set, thereby solving the problems of large memory consumption and insufficient training of few sample categories during image processing model training, and improving the training reliability.

Description

Technical Field

[0001] This application relates to the field of image processing model training technology, and in particular to an image processing model training method, an image processing method, and an electronic device. Background Technology

[0002] Image classification technology has been widely applied to various computer vision tasks, but traditional image processing models face dual challenges in performance and efficiency when dealing with the requirements of large-scale class libraries and low resource consumption. While loss functions based on the normalized exponential function and its variants improve feature discrimination ability, they lead to high resource consumption during training because they need to load all class centers. On the other hand, optimization methods such as random sampling can alleviate memory pressure, but due to the long-tail distribution of data, the sparse tail classes are easily ignored during training, resulting in a decrease in the classification accuracy of image processing models. Summary of the Invention

[0003] This application provides an image processing model training method, an image processing method, and an electronic device to solve the technical problems of high memory consumption and insufficient training of few sample classes during image processing model training in related technologies.

[0004] This application provides a method for training an image processing model, which includes: The process involves: acquiring samples from the current batch of the training image set, each carrying a class label; deduplicating the class labels of each sample in the current batch to obtain a first class set; extracting the first class center vector set corresponding to the first class set from the target class center matrix; forming a second class set using the remaining class labels from the training image set that do not belong to the first class set; selecting the target class from the second class set according to the importance weight of each class; extracting the second class center vector set corresponding to the target class from the target class center matrix; the importance weight is determined based on the correlation between the class center vectors of each class and the corresponding sample feature vectors, and the gradient norm of the class center vectors of each class in a preset number of iterations; merging the first and second class center vector sets to obtain the target class center vector set; calculating the loss value based on the sample feature vectors and the target class center vector set; and updating the image processing model training parameters based on the loss value; repeating the steps from acquiring samples from the current batch of the training image set to updating the image processing model training parameters based on the loss value until the termination condition is met.

[0005] This application also provides an image processing method, which includes: acquiring an image to be processed; inputting the image to be processed into a target image processing model; and outputting an image processing result through the target image processing model, wherein the target image processing model is trained according to an image processing model training method.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the image processing model training method described below.

[0007] The process involves: acquiring samples from the current batch of the training image set, each carrying a class label; deduplicating the class labels of each sample in the current batch to obtain a first class set; extracting the first class center vector set corresponding to the first class set from the target class center matrix; forming a second class set using the remaining class labels from the training image set that do not belong to the first class set; selecting the target class from the second class set according to the importance weight of each class; extracting the second class center vector set corresponding to the target class from the target class center matrix; the importance weight is determined based on the correlation between the class center vectors of each class and the corresponding sample feature vectors, and the gradient norm of the class center vectors of each class in a preset number of iterations; merging the first and second class center vector sets to obtain the target class center vector set; calculating the loss value based on the sample feature vectors and the target class center vector set; and updating the image processing model training parameters based on the loss value; repeating the steps from acquiring samples from the current batch of the training image set to updating the image processing model training parameters based on the loss value until the termination condition is met.

[0008] Unlike related technologies, this application supports distributed training on large-scale data by selecting samples in batches from the training image set, enabling efficient batch processing of massive image data. It removes duplicate category labels from the current batch of samples, avoiding repeated calculations of the same category in a single iteration, reducing redundant operations and lowering the computational cost per iteration. It extracts the first set of class center vectors corresponding to the category labels of the current batch of samples from the target class center matrix, ensuring that the gradient direction of backpropagation points to the correct feature space location, preventing the convergence direction of the image processing model from deviating due to missing class centers corresponding to sample category labels. A second set of category labels, consisting of the remaining category labels from the training image set that do not belong to the first category set, serves as a candidate pool for sampling, providing a basis for sampling and ensuring that the image processing model obtains sufficient categories for comparative learning in each iteration. Based on the relationship between the class center vectors of each category and the sample feature vectors... The importance weights are determined by the connectivity and the gradient norm of the class center vectors over a predetermined number of iterations. Based on these weights, classes with high training value for the image processing model are preferentially sampled from the second class set, allowing these high-value classes to participate in loss calculation and enhancing the inter-class discrimination ability of the image processing model. The first and second class center vector sets are merged to form a complete set of loss calculation objects, enabling the image processing model to simultaneously optimize the features of similar samples to move closer together and the features of different categories to move further apart in each iteration. Loss is calculated based on the target class center vector set, requiring only a portion of the class centers to be loaded in each iteration instead of all, significantly reducing memory usage and computational consumption. Co-optimization between the feature extraction network and class centers is achieved by updating network parameters and the sampled class center vectors. This process is repeated until the termination condition is met, allowing the image processing model to gradually converge, ultimately improving the accuracy and training reliability of the image processing model. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A schematic flowchart illustrating an image processing model training method provided in an embodiment of this application; Figure 2 This is a schematic diagram of an image processing model structure provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an image processing model training device provided in an embodiment of this application; Figure 4 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] Image classification technology plays a crucial role in key scenarios such as public safety, identity authentication, and financial payments. With the popularization of edge computing and the development of lightweight smart terminals, the industry has put forward a triple requirement for the technology: compatibility with large-scale category libraries, high-precision recognition, and low resource consumption. For example, city-level surveillance needs to support real-time comparison of millions of categories, and mobile devices need to achieve high-accuracy recognition with limited memory and computing power. Traditional image processing models struggle to balance performance and efficiency.

[0015] In related technologies, a balanced sampling rate is determined and normalized based solely on the number of samples in each category, the number of categories, and preset weighting parameters. This rate is then used to determine the target number of samples for random sampling without replacement. This approach cannot accurately adapt to the complex sample distribution characteristics of large-scale datasets. For example, in large-scale face datasets, a common long-tail distribution exists: a few head identities have a large sample size but a small percentage of total identities; most long-tail identities have a small sample size but a high percentage of total identities. Furthermore, the sampling strategy does not differentiate based on this distribution characteristic. During random sampling, head identities, due to their large sample size, can be adequately trained even with a low sampling rate, while long-tail identities, due to their small sample size, may not be adequately trained even with a relatively high sampling rate, potentially leading to multiple missed samplings and severely impacting the image processing model's accuracy in recognizing long-tail identities, thus failing to achieve identity fairness. Using random sampling without replacement during training on large-scale datasets results in significant resource waste because it does not effectively assess sample importance. During sampling, samples with minimal impact on image processing model performance may be frequently selected, while crucial samples may be missed.

[0016] Furthermore, the loss function in related technologies optimizes the inter-class decision boundary by introducing an angular interval in the angle space between features and class centers, significantly improving feature discrimination ability. However, this framework suffers from severe resource bottlenecks when training on large-scale datasets (number of identities N>107): the number of class centers in the fully connected layer must be exactly the same as N, and each loss calculation requires loading all N class centers, superimposing feature matrices, gradient caches, and other data, which easily exceeds GPU memory limits and leads to training interruption; at the same time, full computation consumes a lot of computing power, and the training cycle often exceeds one week, restricting development efficiency.

[0017] To reduce resource consumption, related technologies propose a class center sampling approach, where only a subset of class centers are selected for computation in each iteration. While early random sampling reduced memory usage, it couldn't adapt to the long-tail distribution characteristics of large-scale datasets: a few top identities, though only about 20% of the total identities, accounted for over 90% of the sample size; most long-tail identities, though about 80% of the total identities, had a sample size of less than 10%. When random sampling filters by identity count with equal probability, top identities, while having a lower probability of selection (approximately 20%), have sufficient sample size per identity, allowing for rapid training completion even if they are not selected in a single iteration. Conversely, long-tail identities, while having a higher probability of selection, have extremely small sample sizes per identity. If they are not selected in a few iterations, insufficient feature learning may result in a long-term lack of training data, ultimately leading to lower recognition accuracy for long-tail identities compared to top identities in the image processing model.

[0018] To address the aforementioned issues, this application employs a class center importance weight quantification (integrating correlation and loss contribution) combined with a sampling bias correction mechanism to prioritize class centers critical to the performance of the image processing model for computation. This approach reduces memory and computational consumption while avoiding insufficient training of long-tail identities, achieving the triple goals of efficient training, high-precision recognition, and identity fairness on large-scale datasets.

[0019] like Figure 1 As shown, one embodiment of this application provides an image processing model training method, which specifically includes the following steps: S1: Obtain the samples of the current batch in the training image set, with each sample carrying a class label.

[0020] First, obtain the training image set, which can contain multiple samples. These samples can be image samples, each carrying a corresponding category label. Image samples can be image data in any format, such as RGB color images, grayscale images, or infrared images.

[0021] Training image sets can be obtained in various ways. For example, they can be read directly from public datasets; they can also be collected in real-world scenes using image acquisition devices such as cameras and surveillance cameras, and then labeled with category labels manually or automatically; new image samples can also be generated by performing data augmentation operations such as rotation, scaling, cropping, and color transformation on existing images to expand the amount of training data.

[0022] Category labels are used to identify the category to which each image sample belongs, such as an identity ID in face recognition or an object category in image classification. All samples in the same category share the same category label. During the training of an image processing model, the category label is used to determine the positive class center of each sample, that is, the class center vector corresponding to the sample's category label.

[0023] The training image set in this application is determined by the image processing model to be trained. The image processing model in this application can be any feature extraction model or classification model based on a deep neural network. The structure and type of the image processing model can be selected and adjusted according to the actual application scenario. The following are several specific examples to illustrate the application scenario of this application, but it should be understood that these examples do not constitute any limitation on the scope of protection of this application.

[0024] The image processing model in this application can be a face recognition network, such as ResNet50 (a residual network) or MobileNet (a lightweight convolutional neural network architecture). Correspondingly, the training image set can include multiple face image samples, each carrying a corresponding identity label. Furthermore, after training the face recognition network based on the aforementioned training image set, the image processing model can be used to perform identity recognition or verification on input face images.

[0025] The image processing model in this application can be an image classification network, such as ResNet (a residual network), EfficientNet (a convolutional neural network that improves efficiency through compound scaling), or Vision Transformer (a visual model based on a self-attention mechanism). Correspondingly, the training image set can include multiple image samples and their corresponding category labels, such as animals, plants, and vehicles. Furthermore, after training the image classification network based on the aforementioned training image set, the image processing model can be used to classify input images.

[0026] The image processing model in this application can be an object detection network, such as YOLO (a single-stage object detection network), Faster R-CNN (a two-stage object detection network based on region proposal), SSD (a single-stage object detection network), etc. Correspondingly, the training image set can include multiple image samples and their corresponding category and location labels. Furthermore, after training the object detection network based on the above training image set, the image processing model can be used to detect and recognize target objects in the input image.

[0027] The image processing model in this application can be a semantic segmentation network, such as UNet (an image segmentation network with a U-shaped symmetric structure), DeepLab (an image semantic segmentation network based on dilated convolution), SegFormer (an image semantic segmentation network based on Transformer), etc. Correspondingly, the training image set can include multiple image samples and their corresponding pixel-level category labels. Furthermore, after training the semantic segmentation network based on the above training image set, the image processing model can be used to classify each pixel in the input image.

[0028] The image processing model in this application can be a medical image analysis network, such as a convolutional neural network or a Transformer. Correspondingly, the training image set can include multiple medical image samples, such as CT scans and X-rays, along with corresponding lesion category labels or diagnostic labels. Furthermore, after training the medical image analysis network based on the aforementioned training image set, the image processing model can be used to detect lesions or assist in diagnosis of input medical images.

[0029] The image processing model in this application can also be a remote sensing image classification network, such as CNN (Convolutional Neural Network) or ResNet. Correspondingly, the training image set can include multiple remote sensing image samples, such as satellite images, aerial images, etc., and corresponding land cover category labels, such as buildings, farmland, water bodies, forests, etc. Furthermore, after training the remote sensing image classification network based on the aforementioned training image set, the image processing model can be used to classify and identify land cover categories in input remote sensing images.

[0030] The image processing model in this application can be a defect detection network, such as CNN or ResNet. Correspondingly, the training image set can include multiple industrial product image samples and corresponding defect category labels, good product labels, and defective product labels. Furthermore, after training the defect detection network based on the above training image set, the image processing model can be used to perform defect detection or quality assessment on the input industrial product images.

[0031] This application uses an image processing model, a face recognition network, and a training image set, including multiple face image samples, as examples for description.

[0032] Training image sets typically contain massive amounts of facial image samples, such as images of hundreds of thousands, millions, or even tens of millions of individuals. The facial images in the training image set are grouped by identity (ID), and each group is then assigned a category label. The training samples for the face recognition network need to conform to certain specifications. Generally, the input image for a face recognition network is a normalized facial image preprocessed using a face alignment algorithm, including a three-channel (R, G, B) color image. In this application, the facial images in the training image set can be fixed-size three-channel (R, G, B) color images, where the image width is 112 pixels, the image height is 112 pixels, and the number of channels is 3. When the input image size is not as described above, this application can perform simple expansion to adapt to changes in the input image size. When the input image is a single-channel grayscale image or an image in other formats (e.g., YUV format), this application can also perform simple expansion to adapt to changes in the input image (channels).

[0033] Before training the image processing model, the total number of training cycles can be set, for example, to 20 cycles. To avoid the order of samples in the dataset affecting the performance of the image processing model, and to improve the generalization performance of the image processing model, at the beginning of each training cycle, all samples in the training image set are randomly rearranged, i.e., their order is shuffled. In each training cycle, the shuffled training image set is divided into smaller sets; this smaller set of samples is called a mini-batch. Depending on the configuration of the training server, the number of samples in each mini-batch can be set to 128, 256, or 512, etc.; preferably, the number of samples in a mini-batch is at least 1024.

[0034] After one training cycle ends, the image processing model training begins the next training cycle. Training ends when the total number of training cycles reaches a preset value. It is worth noting that, in special cases, the total number of samples in the dataset cannot be divided into integer batches within a training cycle. Individual samples may not be used (discarded) or may be used multiple times (to complete the dataset, e.g., adding several used samples to the end of the dataset). The impact of these individual cases on the training results can be ignored.

[0035] After determining the training image set, the training image set can be input into the image processing model to extract the feature vectors of each sample in the training image set; the mean of the feature vectors of each category is determined based on the feature vectors of all samples corresponding to each category in the training image set; the class center vectors corresponding to each category are determined based on the mean of the feature vectors; and the class center vectors are normalized to obtain the target class center matrix.

[0036] For example, suppose we define the face image with index i as I. i Face image I i The input is fed into a pre-defined face recognition network (image processing model), and after forward propagation, it outputs a feature vector f of dimension D. i ,in R D This means that the value in each dimension of this vector is a real number, and there are a total of D dimensions (512 dimensions). The default feature dimension D=512 (balancing feature expressiveness and memory consumption). The feature vector needs to be L2 normalized, and the formula is: ; in (To prevent the denominator from being 0) Represents a normalized face image I i The corresponding feature vector.

[0037] In the initial stage of training the image processing model, the training image set is traversed first, and the number of samples n for each identity (category) j in the training image set is counted. j The mean of the normalized feature vectors of all samples for that identity is calculated as the initial approximation of the true class center (TCC), denoted as . The class center vector w corresponding to identity j j ,by The initial values ​​are used (which are closer to the ideal class centers than random initialization, accelerating convergence), followed by L2 normalization, as shown in the formula: ; The normalized class center vectors of all categories constitute the target class center matrix. (N is the total number of identities in the dataset, here) After L2 normalization, the facial feature vector With class center vector The modulus length of both is 1. The included angle between them is... The cosine value can be directly represented by the vector dot product, i.e. No additional calculation of vector magnitude is required, greatly simplifying the similarity calculation process and improving the training and inference efficiency of image processing models.

[0038] In this application, before training the image processing model, an initialization operation of the target class center matrix is ​​required. Specifically, the training image set is input into the feature extraction network to extract the feature vectors of each image sample in the training image set. For each category in the training image set, the feature vectors of all image samples corresponding to that category are obtained, and the mean of these feature vectors is calculated as the initial value of the class center vector for that category. Then, the class center vectors of each category are normalized to make their magnitudes uniform, resulting in the target class center matrix. Each column in the target class center matrix corresponds to the class center vector of a category, which is used for the extraction and updating of class centers in subsequent training processes. In each subsequent training iteration, the image samples of the current batch are input into the feature extraction network to extract the feature vectors of each image sample in the current batch. The feature vectors are then normalized to make their magnitudes consistent with those of the class center vectors, allowing the similarity between the image samples and each class center to be calculated directly through the vector dot product.

[0039] S2: Deduplicate the class labels of each sample in the current batch to obtain the first class set, and extract the first class center vector set corresponding to the first class set from the target class center matrix.

[0040] The class labels of each sample in the current batch are deduplicated to obtain the first class set. Each class label in the first class set has at least one corresponding sample in the current batch. From the target class center matrix, the class center vectors that match the class labels in the first class set are selected to form the first class center vector set.

[0041] The current batch of training images contains multiple image samples, each carrying a class label. Since multiple samples in the same batch may belong to the same class (for example, a batch may contain multiple images of faces belonging to Zhang San), it is necessary to deduplicate the class labels of each sample in the current batch to extract all unique class labels, forming a first class set. Each class label in the first class set corresponds to at least one sample in the current batch, meaning that this class appears in a sample within the current batch.

[0042] For example, if the current batch contains 5 samples with category labels "A, B, A, C, B", where category A has two corresponding samples, category B has two corresponding samples, and category C has one corresponding sample, then the first category set obtained after deduplication is "A, B, C".

[0043] After determining the first category set, using the category labels in the first category set as indices, class center vectors matching each category label are selected from the pre-constructed target class center matrix, and these class center vectors are combined to form the first class center vector set. The target class center matrix contains the class center vectors corresponding to each category in the training image set, with each column corresponding to the class center vector of one category. The category label uniquely determines its position in the matrix. Each class center vector in the first class center vector set corresponds one-to-one with each category label in the first category set. For example, if the first category set is "A, B, C", then the class center vectors corresponding to categories A, B, and C are extracted from the target class center matrix to form the first class center vector set.

[0044] In some implementations, if all samples in the current batch belong to the same category (i.e., the first category set contains only one category after deduplication), then the first class center vector set contains only one class center vector.

[0045] S3: Take the category labels in the training image set that do not belong to the first category set and form the second category set. Select the target category from the second category set according to the importance weight of each category. Extract the second category center vector set corresponding to the target category from the target category center matrix. The importance weight is determined according to the correlation between the category center vector and the sample feature vector corresponding to each category, and the gradient norm of the category center vector in the preset number of iterations.

[0046] Based on the category labels in the training image set, a complete category set is formed. Then, the category labels from the first category set are removed from the complete category set, and the remaining category labels that do not belong to the first category set are formed into a second category set. In other words, the second category set is the set of all categories in the training image set except for those involved in the current batch of samples. Category labels in the second category set that do not appear in the current batch serve as a candidate pool for negative class sampling in this iteration.

[0047] For example, suppose there are 10 categories (ID1~10) in the training image set. After deduplication in the current batch, the first category set is {ID1, ID2, ID3}, and the second category set is {ID4, ID5, ID6, ID7, ID8, ID9, ID10}, which is the remaining categories in the training image set that do not belong to the first category set.

[0048] The importance weight of each category is calculated based on the similarity between the class center vectors of each category and the sample feature vectors corresponding to each category, and the gradient norm of the class center vectors of each category in a preset number of iterations. The importance weights of each category in the second category set are used as the sampling frequency to sample categories from the second category set. From the target class center matrix, the class center vectors that match the sampled categories are selected to form the second class center vector set.

[0049] Specifically, for each category, the mean of the inner product of its corresponding class center vector and the feature vector of its corresponding sample is calculated to obtain the class center correlation degree for each category. That is, it is determined by calculating the mean of the inner product of the class center vector and the feature vectors of all samples in that category. The larger the value, the closer the class center is to the sample features, and the more representative the class center of that category is to its category. For each category, the mean of the gradient norm of its corresponding class center vector in the most recent preset number of iterations is calculated to obtain the class center loss contribution degree for each category. The importance weight of each category is calculated based on the class center correlation degree and the class center loss contribution degree. The gradient norm is the mean of the gradient norm of the class center vector in the most recent T iterations, where T is a preset positive integer (e.g., T=10) used to smooth historical gradient fluctuations. The larger the gradient norm, the more significant the contribution of a small adjustment of the class center to the loss reduction.

[0050] The importance weights of each category are calculated based on the correlation between the class center vectors of each category and the feature vectors of the corresponding samples (i.e., the similarity between the class center and the samples of its category), and the gradient norm of the class center vectors of each category in a preset number of iterations (i.e., the sensitivity of the class center to changes in loss).

[0051] In some implementations, the correlation degree and gradient norm are normalized separately and then fused (e.g., multiplied and then normalized again) to obtain the importance weights of each category, so that the sum of the importance weights of all categories is 1. In the initial training phase, since historical gradient information has not yet been accumulated, the importance weights can be determined solely based on the correlation degree.

[0052] In this application, the importance of the class center vector is quantified by two core indicators: the effectiveness of closely approximating the true class center (TCC) and the practicality for loss reduction. The importance weight p is obtained by fusing the two indicators. j (satisfy ).

[0053] ; in, F represents the normalized class center vector corresponding to identity j. j It is the set of all sample feature vectors for the j-th identity (already normalized). s represents the total number of samples for the j-th identity.j Indicates class centrality degree. A larger value indicates that the class center is closer to the TCC. Describe set F j any sample feature vector in the dataset.

[0054] In this application, the influence of class centers on loss optimization is measured by the mean of the gradient norms of the most recent T iterations; that is, the contribution of class centers to the loss is calculated as g. j The specific calculation formula is as follows: ; Where T represents the window size for the most recent iteration, T=10 (default, to avoid interference from historical gradients), L t Let be the loss for the t-th iteration. Represents the class center vector The gradient L2 norm, g j The larger the value, the more significant the contribution of small adjustments to the class center to the reduction of loss.

[0055] s j With g j After being normalized by Softmax, the weights are multiplied to obtain the importance weight p of the j-th identity. j The specific formula is as follows: ; Where k is the summation index, a temporary variable used when traversing all categories, k = 1, 2, 3...N, and N is the number of all categories in the training image set. This represents the sum of the weight factors for all categories, used to assign p... j Normalize to a probability distribution, It is the Softmax function (which degenerates into the Sigmoid normalized form in the case of a single variable).

[0056] In the initial stage, when there are no historical gradients, the weights are measured solely by the correlation, as shown below: ; In this application, the total sampling rate is set to r = 0.1~0.3 (default r = 0.2), and the total number of class centers for each sample is S = max(r × N, B), where max() ensures that the number of samples is not less than the batch size B, avoiding loss calculation errors caused by insufficient negative class center vectors (the set of second class center vectors). For example, N = 10 5 At that time, S = 2 × 10 4 It is only 20% of the total number of class centers.

[0057] After determining the total number of samples S, the importance weights p of each category in the second category set are used. jAs the sampling probability, categories are sampled from the second category set to ensure that the final number of class centers participating in the loss calculation reaches S, and all positive class center vectors (from the first category center vector set) of the current batch are retained. Sampling is performed with replacement, meaning that the same category can be selected repeatedly in a single sampling. The sampling results are deduplicated. If the number of deduplicated categories is insufficient for the preset sampling number, the next most important category is selected from the second category set until the preset sampling number is reached.

[0058] Then, from the target class center matrix, class center vectors that match the sampled classes are selected to form a second class center vector set. Each class center vector in the second class center vector set corresponds one-to-one with each sampled class. In this application, by using the second class set as a negative class candidate pool, sampling is only performed from classes not yet covered in the current batch, avoiding duplicate sampling of classes already present in the current batch. Simultaneously, the importance weight integrates two dimensions: the class center's representational ability (association) and its training value (gradient norm), ensuring that the sampled negative classes are the most valuable for training the image processing model, thus reducing computational cost while guaranteeing training effectiveness.

[0059] Unlike related technologies, this application solves the technical problem of incomplete optimization direction and inability of image processing models to effectively learn inter-class decision boundaries in a single iteration by merging the first set of center vectors corresponding to the category labels of the current batch of samples with the second set of center vectors obtained by sampling according to importance weights to form the target class center vector set. The first set of center vectors provides positive class supervision signals, enabling samples of the same class to aggregate in the feature space; the second set of center vectors provides negative class supervision signals, enabling samples of different classes to separate in the feature space. The two work together to enable the image processing model to obtain complete optimization directions of intra-class aggregation and inter-class separation in a single iteration, enhancing the inter-class discrimination ability of the image processing model and improving recognition accuracy.

[0060] S4: Merge the first class center vector set and the second class center vector set to obtain the target class center vector set. Calculate the loss value based on the sample feature vector and the target class center vector set, and update the image processing model training parameters based on the loss value.

[0061] Specifically, the first set of class center vectors contains the class center vectors corresponding to the class labels of all samples in the current batch. This ensures that the class label of each sample in the current batch can participate in the loss calculation, guaranteeing that the image processing model obtains the correct gradient update direction and avoiding deviation in the convergence direction of the image processing model due to the absence of positive classes. The second set of class center vectors contains the class center vectors corresponding to the categories sampled from the second set (the remaining categories not involved in the current batch) according to their importance weights. This is used to introduce the most valuable negative class information for training under limited computational resources, enhancing the inter-class discrimination ability of the image processing model. The merging operation combines the class center vectors from the two sets into a complete set of class center vectors, namely the target set of class center vectors. This set contains both the class center corresponding to the class label of each sample in the current batch and the class center corresponding to the high-value category selected from the remaining categories according to their importance weights, constituting the complete set of class centers required for the loss calculation of the current batch.

[0062] In this way, the image processing model can be optimized simultaneously based on both positive and negative class samples in the loss calculation of each iteration: on the one hand, the similarity between the sample features and the class centers corresponding to their class labels is maximized (intra-class aggregation); on the other hand, the similarity between the sample features and other class centers (including the positive class centers of other samples in the same batch, as well as the class centers sampled from the second class set) is minimized (inter-class separation). The target class center vector set obtained after merging is the final object participating in the loss calculation of this batch, and is subsequently used to calculate the loss value with bias correction.

[0063] After obtaining the target class center vector set, the loss value is calculated based on the feature vectors of each sample in the current batch and the target class center vector set. Specifically: for each sample, the first similarity between its corresponding sample feature vector and the center vectors of each class in the target class center vector set is calculated, and the first correction parameter is determined based on the first similarity and importance weight; for each sample, the second similarity between its corresponding sample feature vector and the other class center vectors in the target class center vector set is calculated, and the second correction parameter is determined based on the second similarity and importance weight; the loss value is calculated based on the first correction parameter, the second correction parameter, and the loss function.

[0064] For each sample in the current batch, the similarity (e.g., cosine similarity) between the sample's feature vector and the class center vectors of each class in the target class center vector set is first. The similarity between the sample and the class center vector corresponding to its class label is the first similarity (positive class similarity), and the similarity between the sample and other class center vectors is the second similarity (negative class similarity). Then, a first correction parameter is determined based on the first similarity and the importance weight of the class center; a second correction parameter is determined based on the second similarity and the importance weight of each class center.

[0065] Specifically, the positive class similarity is divided by the importance weight of the class center to obtain the first correction parameter (i.e., the corrected positive class similarity); each negative class similarity is divided by the importance weight of its corresponding class center to obtain the second correction parameter (i.e., the corrected negative class similarity). By dividing by the importance weight, the sampling bias caused by non-uniform sampling is eliminated, ensuring that the expected value of the loss is consistent with the loss value calculated from all class centers, thus ensuring the stability and accuracy of the image processing model training.

[0066] Then, the loss value is calculated based on the first correction parameter, the second correction parameter, and the preset loss function. Specifically, the first correction parameter (corrected positive class similarity) is exponentially calculated, and each of the second correction parameters (corrected negative class similarity) is exponentially calculated and summed. The loss contribution value of the sample is determined by the ratio of the exponential values ​​of the first correction parameter to the sum of the exponential values ​​of the second correction parameters. The loss contribution values ​​of all samples are averaged and negatively taken to obtain the loss value of the current batch.

[0067] Backpropagation is performed based on the calculated loss value to calculate the gradient of the loss value with respect to each parameter of the feature extraction network, and the gradient of the loss value with respect to the sampled class center vectors in the target class center vector set. Then, the parameters of the feature extraction network and the sampled class center vectors are updated according to the preset learning rate to complete the training of the current batch.

[0068] By merging the first and second class center vector sets to obtain the target class center vector set, the image processing model simultaneously optimizes intra-class aggregation (maximizing positive class similarity) and inter-class separation (minimizing negative class similarity) in each iteration. By introducing a correction mechanism that divides by importance weight in the loss calculation, bias caused by sampling is eliminated, making the loss value calculation more accurate and ensuring the stability and convergence of the image processing model training. Furthermore, since only the target class center vector set needs to be loaded instead of all class centers, GPU memory usage and computational overhead are significantly reduced.

[0069] The loss function is expressed as: ; Among them, L ArcFace-Importance B represents the loss value, and B represents the number of samples in the current batch. Let represent the angle between the feature vector of the i-th sample and the class center vector of the corresponding class label. This represents the class centers in the target class center vector set other than the class center corresponding to the class of the i-th sample, where m represents the angular interval. Let represent the angle between the feature vector of the i-th sample and the class center vector of the j-th class. y represents the number of samples in the set of center vectors of the second class. i represents the class label of the i-th sample; s represents the feature scaling factor, used to strongly distinguish the numerical similarity between different classes. p represents the importance weight of the class of the i-th sample. j This represents the importance weight corresponding to the j-th class center vector in the target class center vector set. Indicates the first similarity. Indicates the second similarity. This represents the first correction parameter, which is the corrected cosine value of the positive class similarity angle (positive class similarity term). This represents the second correction parameter, which is the corrected cosine value of the negative class angle (negative class similarity term).

[0070] The loss function in this application uses the ArcFace loss function as the basic loss function, and introduces an angular interval in the angle space between the feature and the class center. This can effectively amplify the differences between different identity categories. The original loss formula (with all class centers participating) is as follows: ; Among them, L ArcFace B represents the loss value, and B represents the number of samples in the current batch. Let represent the angle between the feature vector of the i-th sample and the class center vector of the corresponding class label. y represents the angle between the feature vector of the i-th sample and the class center vector of the j-th class. i represents the class label of the i-th sample; s represents the feature scale factor, used to strongly distinguish the numerical similarity between different classes; m represents the angular interval; and N represents the total number of classes in the dataset.

[0071] The existing ArcFace loss function requires loading all class centers for loss calculation. In large-scale class scenarios, a full class center matrix needs to be maintained, and the similarity between the features of the current batch of samples and all class centers needs to be calculated in each iteration. When the total number of classes reaches millions, the class center matrix alone can consume a large amount of GPU memory. After adding the feature matrix and gradient cache, it is very easy to exceed the GPU memory limit. At the same time, the full calculation consumes a lot of computing power, and the training cycle often exceeds one week, which seriously restricts the development efficiency.

[0072] Therefore, compared with related technologies, this application introduces a weight correction term into the loss function. By dividing the first similarity term by the importance weight of the first class center and dividing each second similarity term by the importance weight of the corresponding second class center, it achieves the condition that the expected loss calculation after sampling is equivalent to the loss calculation of the full set of class centers. This allows loss calculation to be completed by loading only a portion of the class centers, reducing memory usage and computational load, and making large-scale class training possible with limited GPU resources. At the same time, the biased estimation caused by non-uniform sampling is corrected by dividing by the weight, ensuring the accuracy of loss calculation and ensuring that the model converges along the correct gradient direction. High-value class centers are sampled first due to their high weight, while low-value class centers are sampled less due to their low weight. Computational resources are dynamically focused on the class centers that are most valuable to the model at present, significantly improving training efficiency. Tail classes naturally receive higher sampling priority because their class centers have not yet fully converged and their loss contribution is high. The training sufficiency of tail classes is effectively guaranteed, and the model's recognition accuracy for tail classes is significantly improved, achieving class fairness. Ultimately, this reduces resource consumption while improving the overall model accuracy and training reliability.

[0073] The training parameters of the image processing model include the image processing model parameters and the class center vectors in the target class center vector set. Backpropagation is performed based on the loss value to calculate the gradient of the loss value with respect to the image processing model parameters and the gradient of the loss value with respect to each class center vector in the target class center vector set. The image processing model parameters are updated based on the gradient of the image processing model parameters. The class center vectors in the target class center vector set are updated based on the gradient of each class center vector in the target class center vector set.

[0074] Specifically, the gradient of the image processing model parameters is calculated by differentiating the loss value with respect to the parameters; a preset learning rate is obtained, and the product of the corresponding gradient and the preset learning rate is subtracted from each image processing model parameter to obtain the updated image processing model parameters. This process follows the standard stochastic gradient descent (SGD) algorithm, and by continuously adjusting the network parameters, the loss value gradually decreases, and the image processing model gradually converges to the optimal state.

[0075] The gradient of each class of center vectors in the target class center vector set is calculated by taking the derivative of the loss value with respect to the center vectors of each class in the target class center vector set. The gradient norm is calculated based on the gradient of each class of center vectors in the target class center vector set, and the gradient norm is stored in the gradient cache. The gradient norm stored in the gradient cache is deleted so that the gradient cache stores the gradient norm for a preset number of times. The importance weight of each class is updated according to the gradient norm stored in the gradient cache for use in the next batch of sampling.

[0076] For updating the class center vectors, the gradient of each class center vector is calculated by differentiating the loss value with respect to the class center vectors in the target class center vector set. Then, the corresponding class center vectors are updated according to the gradients of each class center vector, with the formula: Updated class center vector = Current class center vector - Preset learning rate × Gradient of the class center vector. After updating, normalization is re-performed to ensure that the magnitude of the class center vectors remains uniform.

[0077] After each batch of training is completed, the gradient norm corresponding to each sampled class center vector in the target class center vector set is calculated. The gradient norm is the L2 norm of the gradient vector, which is the square root of the sum of the squares of the elements of the gradient vector. It is used to characterize the sensitivity of the class center to changes in loss in the current batch.

[0078] The calculated gradient norm is stored in the gradient cache corresponding to the class center, and the oldest gradient norm stored in the gradient cache is deleted, so that the gradient cache always contains the gradient norm of the most recent preset number of iterations (e.g., the most recent T=10 iterations). The gradient cache maintains a fixed-length queue for each class center to store the gradient norm of that class center in the most recent number of sampling iterations.

[0079] Then, the importance weights of each category are recalculated based on the gradient norms stored in the gradient cache for use in the next batch of sampling. Specifically, for each category, the mean of all gradient norms in its gradient cache is calculated as the loss contribution of that category; then, the correlation between the class center vector of that category and the feature vectors of the samples under the corresponding category is combined, and the correlation and loss contribution are normalized separately and then fused (e.g., multiplied and then normalized again) to obtain the updated importance weights. In this way, as the training process progresses, the importance weights are continuously adjusted according to the real-time state of the image processing model, enabling the sampling strategy to dynamically adapt to the actual importance of each category at different stages of training.

[0080] S5: Repeat the steps of obtaining samples from the current batch in the training image set and updating the training parameters of the image processing model according to the loss value until the termination condition is met.

[0081] Termination conditions can be one or a combination of several of the following: reaching a preset number of training epochs, reaching a preset number of iterations, or the image processing model no longer showing improved performance on the validation set (early stopping mechanism).

[0082] By repeatedly executing the steps of obtaining samples from the current batch in the training image set and updating the training parameters of the image processing model through backpropagation until the termination condition is met, the image processing model gradually converges during iterative training, ultimately obtaining a feature extraction image processing model with good class discrimination ability. Simultaneously, a gradient caching mechanism dynamically records the sensitivity of each class center to changes in loss and updates the importance weights in real time accordingly. This allows the sampling strategy to adaptively adjust during training. In the early stages of training, class centers with larger gradient norms receive higher weights, accelerating the convergence of the image processing model. In the later stages of training, class centers with smaller gradient norms are also continuously updated, preventing tail classes from being ignored for a long time. This effectively solves the problem of insufficient training of tail classes under long-tailed distributions, and overall improves the classification accuracy and training reliability of the image processing model.

[0083] In one embodiment, this application further includes a mechanism that, in response to the completion of training in the current batch, checks the values ​​of update counters corresponding to various class center vectors; if a class center vector is sampled in the current batch, the update counters corresponding to the sampled class center vectors in the current batch are cleared to zero; if a class center vector is not sampled in the current batch, the values ​​of the update counters corresponding to the unsampled class center vectors in the current batch are incremented; if the value of any update counter exceeds a preset threshold, the corresponding class center vector is marked as a class center vector that must be sampled in the next batch, and the marked class center vector is added to the target class center vector set of the next batch.

[0084] To further prevent some class center vectors from becoming outdated or drifting due to prolonged lack of sampling, this application also includes a forced update mechanism. This mechanism involves configuring an update counter for each class center vector in the training image set, initializing all update counters to zero at the start of training. After each batch of training is completed, the values ​​of the update counters corresponding to each class center vector are checked, and the counters are updated accordingly based on whether the class center vector was sampled in the current batch.

[0085] Specifically, if a certain type of center vector is sampled in the current batch (i.e., the center vector belongs to the target class center vector set of the current batch), the update counter corresponding to the center vector is cleared to zero, indicating that the center vector has been updated; if a certain type of center vector is not sampled in the current batch, the update counter corresponding to the center vector is incremented by 1, indicating that the number of times the center vector has not been sampled has increased.

[0086] When the value of any update counter exceeds a preset threshold, the class center vector corresponding to that update counter is marked as a class center vector that must be sampled in the next batch. In the next batch of training, the marked class center vector is forcibly added to the target class center vector set of the next batch. That is, regardless of its importance weight, the class center vector participates in the loss calculation and parameter update of the next batch. After the forced addition is completed, the update counter corresponding to the class center vector is reset to zero and the counting starts again.

[0087] The preset threshold can be set according to factors such as the size of the training image set, the total number of categories, and the batch size. In one embodiment, the preset threshold is 50, that is, if a certain class center vector is not sampled for 50 consecutive iterations, it is forced to participate in the next batch of training.

[0088] In some implementations, the update counter and the sampling of categories from the second category set together constitute two layers of protection: the first layer is a normal sampling mechanism based on importance weights, where categories with high importance weights are sampled first; the second layer is a forced update mechanism, where even if a category has a consistently low importance weight, it will be forcibly added to the target class center vector set after its consecutive unsampled count exceeds a preset threshold, preventing its class center vector parameters from becoming outdated or drifting due to long-term lack of training. This effectively avoids the problem of low-importance-weight but still valuable class center vectors becoming outdated or drifting due to long-term unsampled status. In particular, it ensures that tail categories with sparse sample sizes and consistently low importance weights receive basic update opportunities during training, further improving the image processing model's recognition accuracy and fairness for tail categories.

[0089] Compared to related technologies that sample samples and achieve balance by adjusting the proportion of samples in each category, this application focuses on class center sampling. It quantifies the importance of class centers using two indicators: class center correlation (measuring the closeness of class centers to real class centers) and class center loss contribution (measuring the impact of class centers on loss optimization). The core is to sample after evaluating the training value of class centers, rather than simply adjusting the proportion of samples. This finer sampling granularity allows for precise selection of class centers that are most meaningful for optimizing the image processing model at the current stage, avoiding wasting computational resources on already well-trained categories. This ensures that each batch of sampled class centers effectively drives the image processing model's optimization. Importance weights are dynamically calculated for each batch based on the class center state, reflecting the training value of each type of center in real time, ensuring the sampling strategy remains synchronized with the image processing model's training state. Tail categories, due to their class centers not yet fully converged and their higher loss contribution, naturally receive higher sampling priority. Combined with a forced update mechanism, this provides a double guarantee, effectively improving the training sufficiency of tail categories. Simultaneously, the weight correction term in the loss function makes the expected loss calculation after sampling equivalent to the full calculation, forming a complete closed loop. This reduces memory and computational consumption while ensuring training accuracy, ultimately improving the overall recognition accuracy and class fairness of the image processing model.

[0090] In large-scale image classification training, the training progress and convergence speed of different categories often differ significantly. Head classes, due to sufficient sample size, typically converge fully early in training, their class center vectors stabilize, their gradient norms continuously decrease, and their corresponding importance weights remain at a low level for a long period. Tail classes, on the other hand, with fewer samples, converge more slowly and require continuous training opportunities. If fully converged head classes continue to participate in each batch of sampling and loss calculation, it will lead to unnecessary waste of computational resources and reduce overall training efficiency.

[0091] To address the aforementioned issues, this application sets up a system during image processing model training that records the importance weight of each category across multiple consecutive training cycles. When the importance weight of any category falls below a preset freezing threshold for a predetermined number of consecutive training cycles, it indicates that the category's contribution to improving the image processing model's performance in the current training phase is relatively low, and at this point, the category is marked as a frozen category.

[0092] In subsequent sampling, frozen categories are temporarily excluded from the second category set. This means frozen categories are no longer sampled from the second category set, nor are they involved in loss calculations or class center vector updates. This avoids the fully trained head categories continuously consuming computational resources, allowing training resources to be concentrated on the tail categories that have not yet fully converged, thereby improving overall training efficiency.

[0093] Meanwhile, to prevent categories from being permanently frozen and losing subsequent training opportunities, this application also sets up a system to recalculate the importance weights of frozen categories every preset number of training cycles (e.g., 3 to 5 training cycles), and determines whether to unfreeze them based on the recalculated importance weights. If the importance weight of a category recovers to above the preset freezing threshold, it is unfrozen, regaining its eligibility to participate in sampling and training. This ensures that when the importance of a frozen category rises again due to factors such as changes in data distribution, it can be promptly reintroduced into training, guaranteeing the long-term adaptability and overall accuracy of the image processing model.

[0094] In some implementations, the preset freeze threshold can be set based on the total number of categories in the training image set, the distribution of sample sizes for each category, and the size of the image processing model. In one specific implementation, the preset freeze threshold is one-tenth of the average importance weight of all categories. If the importance weight of a certain category is lower than this threshold for several consecutive training epochs, it indicates that the training for that category has converged sufficiently, and temporarily freezing it will not affect the overall performance of the image processing model.

[0095] It's important to note that category freezing and the aforementioned forced update are complementary, not conflicting. The freezing mechanism targets categories with consistently low importance weights that have been sufficiently trained, reducing computational redundancy by temporarily excluding them. The forced update mechanism, on the other hand, targets tail-end categories with low importance weights that haven't received an update opportunity for a long time, ensuring their basic training chances by forcibly adding them. Together, they maximize training efficiency while ensuring sufficient training for each category.

[0096] In addition, if a frozen category triggers a forced update condition during the freezing period (i.e., its update counter exceeds a preset threshold), the forced update mechanism will be triggered first, and the category will be forcibly added to the next batch of target class center vector sets, ensuring that no category will completely lose its training opportunity due to freezing.

[0097] Please see Figure 2 In one specific embodiment, this application can use ResNet50 to construct the neural network for face recognition. The neural network for face recognition includes the first five convolutional layers of ResNet50: conv1 (convolutional layer 1), conv2_x (convolutional layer 2), conv3_x (convolutional layer 3), conv4_x (convolutional layer 4), and conv5_x (convolutional layer 5), as well as a first fully connected layer FC1, a second fully connected layer FC2, and a loss layer. The first five convolutional layers of ResNet50, together with FC1, constitute the backbone network (i.e., the face recognition network), used to convert the input face image into a low-dimensional feature vector, which is also called a feature embedding.

[0098] In this embodiment, the input to the FC1 layer is the output feature of the conv5_x layer, and the output of the FC1 layer is a 512-dimensional feature vector. The 512-dimensional feature vector achieves a good balance between feature expressiveness and computational efficiency, ensuring sufficient discriminative power while avoiding the computational and storage overhead caused by excessively high dimensionality. It should be noted that 512 dimensions are merely an exemplary setting; other dimensions, such as 128, 256, or 1024 dimensions, can be selected in other embodiments.

[0099] The input to the FC2 layer is the output feature of the FC1 layer. The dimensionality of the output feature of the FC2 layer is the same as the total number of categories (IDs) in the training set, with one dimension corresponding to one category. Each column of the weight matrix of the FC2 layer corresponds to the class center vector of a category. The class center vector and the input feature participate in the loss calculation. The class center vector is a representative vector of each category in the feature space, used to represent the feature direction of that category during training. Each category corresponds to one class center vector, and all class center vectors together form the class center matrix. The FC2 layer is also used to select the class center vector corresponding to the first category set (the first class center vector set) from the target class center matrix in response to the call of the loss function layer, and to select the class center vector corresponding to the sampled categories (the second class center vector set).

[0100] The loss function layer is used to: calculate the first similarity and the second similarity based on the sample feature vector and the target class center vector set; determine the first and second correction parameters based on the first and second similarities and importance weights; and calculate the loss value based on the first correction parameters, the second correction parameters and the loss function formula.

[0101] The importance weight can be calculated in the loss function layer or by a separate weight calculation module. Specifically, it includes: calculating the correlation degree based on the mean of the inner product between the class center vector of each category and the corresponding sample feature vector; calculating the loss contribution based on the gradient norm of the class center vector in a preset number of iterations; and normalizing and fusing the correlation degree and the loss contribution to obtain the importance weight.

[0102] Before calculating the loss at the loss function layer, in response to a call from the loss function layer, the FC2 layer merges the first class center vector set and the second class center vector set into a target class center vector set, and provides this target class center vector set to the loss function layer for loss calculation.

[0103] Furthermore, this application also adjusts the structure of the conv1 layer of the standard ResNet50. The conv1 layer of the standard ResNet50 is typically a 7×7 convolutional layer with a stride of 2. In this application, to meet the requirements for preserving detailed features in face recognition tasks, the conv1 layer is set to a 3×3 convolutional layer with a stride of 1.

[0104] The neural network structure and parameter settings for building a face recognition network using ResNet50 described above are merely illustrative examples, and this application is not limited to specific network structures and parameters.

[0105] In one specific embodiment, the image processing model training method provided in this application includes a data preprocessing stage and an image processing model training stage, and the specific steps are as follows: 1. Data preprocessing stage: The data preparation for the training phase is completed in the data preprocessing stage. Specifically, firstly, the training image set is traversed, and the training sample size n corresponding to each identity j is counted. j The correspondence between identity IDs and sample sizes is stored in a dictionary (denoted as count). dict ), in the format of , where N is the total number of identities in the dataset.

[0106] Extract all sample features for each identity j and calculate the feature mean. (As an initial TCC approximation), stored as an array ;here .

[0107] Initialize the gradient cache array gard cache[j] (length N) to store the norm of the most recent T gradients for each class center, with an initial value of 0 for each.

[0108] Initialize the target class center matrix W of the fully connected layer (FC2). This matrix has dimensions D×N (where D is the feature dimension, default D=512), and each column corresponds to a class center vector w for an identity. j The initial value is set to Perform L2 normalization on all class centers in W to obtain ; Calculate the initial importance weights (Based solely on correlation degree s) j ), stored as an array .

[0109] 2. The image processing model training phase will be described using a face recognition network as an example: First, set the initial parameters, including the training input, training output, and loss function.

[0110] Training input: Sample data read from the training image set; initial training epoch e=1, total training epochs ; Set the parameters related to the stochastic gradient descent algorithm, such as the initial learning rate L. r =0.1, momentum coefficient β=0.9, weight decay coefficient decay=0.0005, and the learning rate decreases to L in the 10th training cycle. r =0.01, and the learning rate drops to L in the 15th training cycle. r =0.001, and the learning rate drops to L in the 18th training cycle. r =0.0001. The sample size B of a small batch can be 1024.

[0111] Training output: Face recognition network m φ (The image processing model parameter is φ).

[0112] Loss function: Set the loss function to the loss function provided in this application.

[0113] Step 1: Initialize the learnable parameters of the image processing model. The face feature extraction network can be initialized using a random initialization method.

[0114] FC2 layer class center parameters: initialized according to W during the data preprocessing stage; gradient cache array grad cache: Initial value is set to 0; Class center vector update counter cnt j (Length N): The initial value is set to 0 (to record the number of iterations in which the class center vector was not sampled).

[0115] Step 2: When e (current training cycle number) ≤ E (total training cycle number), perform the following operations: a) e = e + 1 (update the current training cycle); b) Randomly shuffle the sample order of the training image set to avoid order dependency affecting generalization performance; c) Batch data loading and feature extraction: Suppose we select a batch of V training samples I1, I2, ..., I from the training image set. v , and their corresponding labels y1, y2, ... y v V training samples are input into the face feature extraction network, and the output sample feature vector F={f1,f2,......f v Then, perform L2 normalization on F to obtain... .

[0116] d) Calculate importance sampling class centers: (d.1) Extracting positive class centers: For labels y1, y2, ... y v After removing duplicates, we obtain the set of positive identities (the first category set) Y. pos-unique This corresponds to the set of positive class center vectors (the set of center vectors of the first class). .

[0117] (d.2) Sampling negative class centers: Negative identity set (second category set) Y neg ={1,2,......N}\Y pos-unique Sampling with replacement is performed according to importance weights, with a sample size of S-|Y. pos-unique |,(|Y pos-unique | represents the number of elements in the positive class identity set); after removing duplicates from the sampling results, we obtain the negative class center set (the set of second class center vectors) W. neg-sample If the number after deduplication is insufficient, S-|Y pos-unique |, supplement the negative class identities with the second highest sampling weight until the number meets the requirement; (d.3) Generate the final set of sampled class center vectors: merge W pos With W neg-sample This yields the set of target class center vectors, which is W. sample =W pos ∪W neg-sample The corresponding set of identity IDs is denoted as J. sample L2 normalization is performed again on all class center vectors in the target class center vector set to ensure uniform magnitude.

[0118] e) Calculate the importance sampling loss: based on (Sample features) and the target class center vector set, calculate the cosine value of the positive class angle (first similarity) for each sample i (1≤i≤V). Calculate the cosine value of the included angle between the negative classes (second similarity). Substitute the values ​​into the loss function to calculate the loss value.

[0119] f) Backpropagation and parameter update: (f.1) Gradient Calculation: The loss is calculated automatically to affect the parameters and W of the face feature extraction network. sample The gradient of the class center vector; (f.2) Update backbone network parameters: Use stochastic gradient descent algorithm (including momentum and weight decay) to update the learnable parameters of the face feature extraction network; (f.3) Update the sampling class center: for W sample class center vectors in Perform the update: After the update, L2 normalization is re-executed; among them, This represents the updated class center vector of the j-th class. Indicates the loss value L for The gradient is a vector whose direction points in the direction of the fastest increase of the loss function, and its magnitude represents the effect of the loss on the loss function. Sensitivity to change.

[0120] (f.4) Reset the class center vector counter: The counter cnt will be updated. j Reset to 0; for unsampled class center vectors Update the counter value to cnt j +1.

[0121] g) Gradient caching and weight updates: (g.1) Update gradient cache: For Calculate the current gradient norm Store it in gard cache[j] and delete the oldest value in the cache (keeping the cache length T=10); (g.2) Recalculate importance weights: For all identities j, recalculate the association degree s. j With contribution to loss g j And recalculate the importance weights to provide a basis for the next batch of sampling.

[0122] h) Force update of unsampled class centers: Check the counter cnt for all unsampled class center vectors j If cnt j >K max (Default K) max (Can be set to 50), forcibly adding this identity to the next batch of Y. pos-unique The corresponding sampling set is used to avoid parameter drift caused by long periods without updates.

[0123] i) Repeat steps c) to h) until all batches of samples in the current cycle have been processed.

[0124] j) End the current training cycle.

[0125] Step 3: Training ends and image processing model output.

[0126] Training is stopped once all E=20 training cycles have been completed.

[0127] This application proposes a dual-indicator quantification and fusion mechanism for class center importance. By combining two indicators—class center correlation (measuring the closeness of a class center to the real class center, calculated based on the mean inner product of the class center and all sample features of that identity) and class center loss contribution (measuring the impact of a class center on loss optimization, calculated based on the mean gradient norm of the most recent T iterations)—after Softmax normalization, a sampling probability weight for class centers is generated. This achieves accurate quantification of the training value of class centers, providing a scientific basis for subsequent sampling. This application also proposes an importance sampling mechanism with bias correction and forced updates. When sampling class centers according to the above weights, a weight correction term (the exponential terms of positive and negative class centers are divided by their corresponding importance weights) is introduced into the loss calculation to offset sampling bias and ensure the accuracy of loss calculation. Simultaneously, an update counter is maintained for class centers, forcibly adding class centers that have not been sampled for a long time (counter exceeding the threshold) to the next batch of sampling, avoiding parameter drift and balancing training efficiency and identity fairness. Thus, this application can reduce the computational resource consumption of large-scale training of face recognition networks while ensuring the sufficiency of training for identities at each sample size level (especially identities with small sample sizes), ultimately improving the training efficiency and recognition accuracy of image processing models.

[0128] It is understood that the execution entity of the image processing model training method provided in this application can be a server or a server cluster, which can be local or cloud-based. Of course, the execution entity of this method can be any computing device capable of implementing the method, and this application does not limit the type of computing device to which the execution entity is.

[0129] In one specific embodiment, the image processing method provided in this application includes: Step 101: Obtain the image to be processed.

[0130] Obtain the image to be processed, which is an image that conforms to the input specifications of the target image processing model. The image to be processed can be a single image or multiple images.

[0131] In one embodiment, the image to be processed is a face image to be identified. If the acquired original image contains a face, it can be processed into a face image that conforms to the input specifications of the target image processing model through image processing operations such as face detection technology, face feature point localization technology, and image affine transformation technology.

[0132] Step 102: Input the image to be processed into the target image processing model, and output the image processing result through the target image processing model. The target image processing model is trained using the image processing model training method provided in this application.

[0133] In one embodiment, the target image processing model can directly output the category corresponding to the image to be processed. In another embodiment, the target image processing model outputs the feature vector of the image to be processed, and external processing logic compares the feature vector with reference features in a preset database to determine the category corresponding to the image to be processed based on the comparison result.

[0134] For example, the image to be processed is input into the target image processing model, and forward propagation is performed through the feature extraction network of the target image processing model to output the feature vector (i.e., feature embedding) of the image to be processed. The target image processing model is trained according to the image processing model training method provided in this application. When there are multiple images to be processed, the target image processing model outputs multiple feature vectors, each feature vector corresponding to one image.

[0135] In one embodiment, the target image processing model is a face recognition model, and the face feature vector output by the feature extraction network is used for subsequent face recognition processing.

[0136] One or more feature vectors extracted from the image to be processed are compared with reference features stored in a pre-set database, and the similarity between the feature vectors and each reference feature is calculated. Based on the similarity calculation results, the category corresponding to the image to be processed is determined, and the image processing result is output.

[0137] In one embodiment, the preset database is a face recognition database to be queried. The face recognition database to be queried stores individual identity identifiers (IDs) corresponding to reference face images and their corresponding reference face feature vectors. The reference face feature vectors in this database are stored after pre-extracting features from the reference face images using the feature extraction network of the target image processing model trained in this application. The face recognition database to be queried contains reference face feature vectors corresponding to at least one identity identifier, and each identity identifier can correspond to one or more reference face feature vectors.

[0138] In one embodiment, the reference face feature vector also corresponds to a weighting coefficient, which is used to indicate the importance of the corresponding reference face feature vector. The weighting coefficient can be the detection score output by the face detector, the score of the degree of face blurring, or the score of the degree of face occlusion; this application does not impose any limitations on this.

[0139] During feature matching, the similarity between the feature vector of the image to be processed and the feature vector of each reference face in the face recognition database to be queried is calculated.

[0140] In some implementations, the similarity can be the cosine similarity between the mean of one or more feature vectors of the image to be processed and the mean of one or more reference face feature vectors corresponding to each identity in the face recognition database to be queried. The mean can be an arithmetic mean or a weighted mean (weighted using weighting coefficients).

[0141] In other implementations, similarity can be the cosine similarity between the feature vector with the highest importance value among one or more feature vectors of the image to be processed and the feature vector with the highest importance value among the feature vectors of each reference face in the face recognition database to be queried. When calculating similarity, feature vectors with low importance can also be excluded based on their importance scores, retaining only the feature vectors with high importance for similarity calculation.

[0142] Based on the comparison results, the category corresponding to the database entry with the highest similarity to the feature vector of the image to be processed is determined as the recognition result of the image to be processed and output.

[0143] In one embodiment, when the target image processing model is a face recognition model, the output is the face identity identifier (ID) in the face database that has the highest similarity to the input face image, which is used as the face recognition result.

[0144] The embodiments of this application provide an image processing model training device, which is specifically as follows: Figure 3 As shown, the image processing model training device includes a processing module 20 and an iteration module 21.

[0145] Processing module 20 is used to acquire samples of the current batch in the training image set, each sample carrying a class label; to deduplicate the class labels of each sample in the current batch to obtain a first class set; to extract the first class center vector set corresponding to the first class set from the target class center matrix; to form a second class set using the remaining class labels in the training image set that do not belong to the first class set; to select the target class from the second class set according to the importance weight of each class; to extract the second class center vector set corresponding to the target class from the target class center matrix; the importance weight is determined based on the correlation between the class center vector of each class and the sample feature vector of each class, and the gradient norm of the class center vector of each class in a preset number of iterations; to merge the first class center vector set and the second class center vector set to obtain the target class center vector set; to calculate the loss value based on the sample feature vector and the target class center vector set; and to update the training parameters of the image processing model based on the loss value.

[0146] Iteration module 21 is used to repeatedly execute the steps of obtaining samples of the current batch in the training image set and updating the training parameters of the image processing model according to the loss value until the termination condition is met.

[0147] Specific limitations regarding the image processing model training device can be found in the limitations of the image processing model training method described above, and will not be repeated here. Each module in the aforementioned image processing model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0148] like Figure 4 As shown, embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described image processing model training method embodiments or the steps in any of the above-described image processing method embodiments.

[0149] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described image processing model training method embodiments or the steps in any of the above-described image processing method embodiments when running.

[0150] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0151] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described image processing model training method embodiments or the steps in any of the above-described image processing method embodiments.

[0152] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described image processing model training method embodiments or the steps in any of the above-described image processing method embodiments.

[0153] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "electronic device," or "apparatus" used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0154] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] The foregoing has provided a detailed description of an image processing model training method and the steps in an embodiment of the image processing method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for training an image processing model, characterized in that, The method includes: Obtain samples from the current batch in the training image set, wherein the samples carry class labels; The category labels of each sample in the current batch are deduplicated to obtain the first category set. The first category center vector set corresponding to the first category set is extracted from the target class center matrix. The remaining category labels in the training image set that do not belong to the first category set are used to form a second category set. The target category is selected from the second category set according to the importance weight of each category. The second category center vector set corresponding to the target category is extracted from the target category center matrix. The importance weight is determined based on the correlation between the category center vector and the sample feature vector corresponding to each category, and the gradient norm of the category center vector in a preset number of iterations. Merge the first set of center vectors and the second set of center vectors to obtain the target set of center vectors. Calculate the loss value based on the sample feature vectors and the target set of center vectors. Update the training parameters of the image processing model based on the loss value. Repeat the steps of obtaining samples of the current batch in the training image set and updating the training parameters of the image processing model according to the loss value until the termination condition is met; The step of selecting the target category from the second category set according to the importance weight of each category, and extracting the second category center vector set corresponding to the target category from the target category center matrix includes: The importance weight of each category is calculated based on the similarity between the class center vector of each category and the sample feature vector corresponding to each category, and the gradient norm of the class center vector of each category in a preset number of iterations. Using the importance weight of each category in the second category set as the sampling frequency, categories are sampled from the second category set; From the target class center matrix, class center vectors that match the sampled categories are selected to form the second class center vector set.

2. The image processing model training method according to claim 1, characterized in that, The process of obtaining samples from the current batch in the training image set includes: The training image set is input into the image processing model to extract the feature vectors of each sample in the training image set. The mean value of the feature vector for each category is determined based on the feature vectors of all samples corresponding to each category in the training image set. The class center vector corresponding to each category is determined based on the mean of the feature vectors. The class center vectors are normalized to obtain the target class center matrix.

3. The image processing model training method according to claim 2, characterized in that, Before performing duplicate removal of category labels for each sample in the current batch, the following steps are included: Input the samples of the current batch into the image processing model to extract the feature vectors of each sample in the current batch; The feature vectors of each sample in the current batch are normalized to obtain the sample feature vectors. The target class center matrix has the same magnitude as the sample feature vectors.

4. The image processing model training method according to claim 1, characterized in that, The process of deduplicating the category labels of each sample in the current batch to obtain a first category set, and extracting the first category center vector set corresponding to the first category set from the target class center matrix includes: The category labels of each sample in the current batch are deduplicated to obtain a first category set, wherein each category label in the first category set corresponds to at least one sample in the current batch; From the target class center matrix, select the class center vectors that match the category labels in the first category set to form the first class center vector set.

5. The image processing model training method according to claim 1, characterized in that, The step of forming a second category set using the remaining category labels from the training image set that do not belong to the first category set includes: Based on the category labels in the training image set, a complete category set is formed; Remove each category label from the first category set from the total category set, and form the remaining category labels that do not belong to the first category set into the second category set.

6. The image processing model training method according to claim 1, characterized in that, The calculation of the importance weight of each category based on the correlation between the class center vectors of each category and the corresponding sample feature vectors of each category, and the gradient norm of the class center vectors of each category in a preset number of iterations, includes: For each category, the mean of the inner product between its corresponding class center vector and its corresponding sample feature vector is calculated to obtain the class center correlation degree of each category; For each category, the mean of the gradient norm of its corresponding class center vector in the most recent preset number of iterations is calculated to obtain the class center loss contribution of each category; The importance weight of each category is calculated based on the class center correlation degree and the class center loss contribution degree.

7. The image processing model training method according to claim 1, characterized in that, The step of calculating the loss value based on the sample feature vector and the target class center vector set includes: For each sample, calculate the first similarity between its corresponding sample feature vector and the center vectors of each class in the target class center vector set, and determine the first correction parameter based on the first similarity and the importance weight; For each sample, calculate the second similarity between its corresponding sample feature vector and other class center vectors in the target class center vector set, and determine the second correction parameter based on the second similarity and the importance weight; The loss value is calculated based on the first correction parameter, the second correction parameter, and the loss function.

8. The image processing model training method according to claim 7, characterized in that, The loss function is expressed as: ； Among them, L ArcFace-Importance B represents the loss value, and B represents the number of samples in the current batch. Let represent the angle between the feature vector of the i-th sample and the class center vector of the corresponding class label. This represents the class centers in the target class center vector set other than the class center corresponding to the class of the i-th sample, where m represents the angular interval and θ represents the other class centers. i,j Let represent the angle between the feature vector of the i-th sample and the class center vector of the j-th class. y represents the number of samples in the set of center vectors of the second class. i represents the class label of the i-th sample; s represents the feature scaling factor, used to strongly distinguish the numerical similarity between different classes. p represents the importance weight of the class of the i-th sample. j This represents the importance weight corresponding to the j-th class center vector in the target class center vector set. Indicates the first similarity. Indicates the second similarity. Indicates the first correction parameter. This indicates the second correction parameter.

9. The image processing model training method according to claim 1, characterized in that, The image processing model training parameters include image processing model parameters and class center vectors in the target class center vector set. Updating the image processing model training parameters based on the loss value includes: Backpropagation is performed based on the loss value to calculate the gradient of the loss value with respect to the parameters of the image processing model and the gradient of the loss value with respect to the center vectors of each class in the target class center vector set. Update the image processing model parameters based on the gradient of the image processing model parameters; Update the class center vectors in the target class center vector set based on the gradients of the class center vectors in the target class center vector set.

10. The image processing model training method according to claim 9, characterized in that, The step of calculating the gradient of the loss value with respect to the image processing model parameters, and updating the image processing model parameters based on the gradient of the image processing model parameters, includes: The gradient of the image processing model parameters is calculated by differentiating the loss value with respect to the image processing model parameters. Obtain the preset learning rate, and update the image processing model parameters based on the gradient of the image processing model parameters and the preset learning rate.

11. The image processing model training method according to claim 9, characterized in that, Calculating the gradient of the loss value with respect to each class center vector in the target class center vector set, and updating the class center vectors in the target class center vector set based on the gradients of each class center vector in the target class center vector set, includes: The gradient of each type of center vector in the target class center vector set is calculated by taking the derivative of the loss value with respect to the center vector of each type in the target class center vector set; The gradient norm is calculated based on the gradients of the various center vectors in the target class center vector set. The gradient norm is stored in the gradient cache, and the gradient norm with the earliest storage time in the gradient cache is deleted so that the gradient cache stores the gradient norm for a preset number of times. Based on the gradient norms stored in the gradient cache, update the importance weights of each category for use in the next batch of sampling.

12. The image processing model training method according to claim 1, characterized in that, The method further includes: In response to the completion of training for the current batch, check the values ​​of the update counters corresponding to various center vectors; If a class center vector is sampled in the current batch, the update counter corresponding to the sampled class center vector in the current batch is cleared to zero. If the class center vector is not sampled in the current batch, increment the value of the update counter corresponding to the unsampled class center vector in the current batch; If the value of any update counter exceeds a preset threshold, the class center vector corresponding to the update counter that exceeds the preset threshold is marked as the class center vector that must be sampled in the next batch, and the marked class center vector is added to the target class center vector set of the next batch.

13. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is input into the target image processing model, and the target image processing model outputs the image processing result. The target image processing model is trained using the image processing model training method according to any one of claims 1 to 12.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the image processing model training method as described in any one of claims 1 to 12 or the steps of the image processing method as described in claim 13 when executing the computer program.

Images