Model optimization method, system, and electronic device

Abstract

The application relates to a model optimization method, system and electronic equipment. The method comprises the following steps: obtaining performance feedback information of a target detection model on a verification data set; determining learning difficulty degrees of the target detection model on each preset category according to the performance feedback information, and determining at least one difficult category from each preset category based on the learning difficulty degrees; determining a synthetic data amount corresponding to each difficult category according to the learning difficulty degrees of each difficult category; generating synthetic samples for each difficult category according to the synthetic data amount, wherein the synthetic samples carry labeled information; and fusing the synthetic samples and real training samples, and updating parameters of the target detection model by using the fused samples. The application can improve the recognition accuracy of the target detection model on the difficult category and the overall detection performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a model optimization method, system, and electronic device. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing technologies, deep learning-based surface defect detection technology has been widely applied in the field of fabric production quality control. Compared with traditional manual visual inspection and machine vision methods based on handcrafted features, deep learning detectors (including object detectors based on convolutional neural networks and end-to-end detectors based on the Transformer architecture) can automatically learn multi-level texture features, achieving high detection accuracy in defect detection tasks on solid-color fabrics and simple texture backgrounds. However, the performance of deep learning detectors is highly dependent on large-scale, class-balanced training data with precise pixel-level or bounding box-level annotations, which requires significant resources for data acquisition and manual annotation before actual deployment.

[0003] Traditional techniques typically employ two types of methods to alleviate the problem of insufficient training data: one is conventional data augmentation methods based on image transformations, such as flipping, rotating, cropping, and color dithering. These methods only interpolate within the existing data manifold and cannot introduce new semantic diversity. The other is synthetic augmentation methods based on generative models, such as generative adversarial networks (GANs) and diffusion models, which generate new defect samples by learning the latent data distribution. GANs have been used for synthesizing fabric defect images, while diffusion models, due to their relatively stable training process and strong image generation capabilities, have been explored in recent years for generating few-sample images of surface defects. Furthermore, to address the problem of long-tailed data distribution, cost-sensitive learning, reweighting strategies, and oversampling methods based on feature space interpolation have emerged, attempting to improve model performance on imbalanced data through algorithmic adjustments or data augmentation.

[0004] However, all of the above-mentioned existing methods have a common fundamental flaw: the generative augmentation process and the learning state of the target detector are independent of each other, forming an open-loop architecture. This makes it impossible for the model to autonomously identify and specifically compensate for its learning deficiencies in each category without the need for human-preset augmentation strategies. Summary of the Invention

[0005] Therefore, it is necessary to provide a model optimization method, system, and electronic device to address the aforementioned technical problems.

[0006] Firstly, this application provides a model optimization method, including: Obtain performance feedback information of the object detection model on the validation dataset; Based on performance feedback information, determine the learning difficulty of the target detection model in each preset category, and based on the learning difficulty, determine at least one difficult category from each preset category; Based on the learning difficulty of each difficulty category, determine the amount of synthetic data corresponding to each difficulty category; Based on the amount of synthetic data, synthetic samples are generated for each difficulty category, and the synthetic samples carry annotation information. Synthetic samples are fused with real training samples, and the parameters of the target detection model are updated using the fused samples.

[0007] In one embodiment, performance feedback information includes the prediction results of the object detection model on the validation dataset; The learning difficulty of the object detection model in each preset category is determined based on performance feedback information, including: Based on the prediction results, the false detection information for each preset category is statistically analyzed, including the false detection rate and / or inter-class confusion rate. The learning difficulty of each preset category is determined based on the false negative rate and / or inter-class confusion rate.

[0008] In one embodiment, the learning difficulty of each preset category is determined based on the false negative rate and / or inter-class confusion rate, including: The total error rate is determined based on the false negative rate, and the inter-class confusion penalty is determined based on the inter-class confusion rate. The learning difficulty of each preset category is determined by the weighted sum of the total error rate of each preset category and the inter-class confusion penalty.

[0009] In one embodiment, the learning difficulty is determined according to the following formula:

[0010] in, This represents the learning difficulty of category c. This represents the correct detection rate for category c. This represents the total error rate. This represents the inter-class confusion rate where the true class c is predicted as class j. denoted by , where C represents the predefined set of categories.

[0011] In one embodiment, based on learning difficulty, at least one difficulty category is determined from a set of preset categories, including: Sort the preset categories according to their learning difficulty from highest to lowest; The categories with the highest sorting counts are designated as the difficult categories; Alternatively, a preset category whose learning difficulty exceeds a preset threshold can be defined as a difficulty category.

[0012] In one embodiment, determining the amount of synthetic data corresponding to each difficulty category includes: Obtain the number of real samples for each difficulty category in the real training set; Based on the actual number of samples, determine the frequency compensation amount for each difficulty category; Based on the degree of learning difficulty, determine the amount of difficulty compensation for each category of difficulty; The amount of composite data for each difficulty category is determined by the weighted sum of the frequency compensation and the difficulty compensation.

[0013] In one embodiment, determining the amount of synthetic data for each difficulty category includes: The frequency compensation amount for each difficulty category is determined based on the difference between the preset reference number and the actual sample number; where the frequency compensation amount is zero when the actual sample number is greater than or equal to the preset reference number. The difficulty compensation amount for each difficulty category is determined based on the product of the learning difficulty and the preset reference quantity. The amount of synthetic data for each difficulty category is determined by the weighted sum of the frequency compensation and the difficulty compensation for each difficulty category.

[0014] In one embodiment, the amount of synthesized data is determined according to the following formula:

[0015] in, This indicates the amount of synthetic data for category c. Indicates the preset reference quantity. This represents the number of true samples in category c. This represents the normalized learning difficulty of category c. Indicates the difficulty compensation intensity coefficient. This represents the set of difficulty categories.

[0016] In one embodiment, based on the amount of synthetic data, synthetic samples are generated for each difficulty category, including: Obtain the background image and its corresponding spatial constraint information; Spatial constraint information is used as a conditional input to control the preset generation model to synthesize sample content that conforms to the corresponding difficulty category features within a specified area of ​​the background image; Based on spatial constraint information, generate annotation information for the sample content.

[0017] In one embodiment, the spatial constraint information includes at least one of the following: a binary mask, a semantic segmentation map, key point coordinates, and bounding box coordinates; Based on spatial constraint information, annotation information for the sample content is generated, including: Perform connectivity analysis on spatial constraint information to determine the boundary information of each connectivity region; Generate corresponding bounding box annotations based on the boundary information of each connected component; Alternatively, the corresponding bounding box annotation can be determined based on the spatial constraint information.

[0018] In one embodiment, the preset generative model includes at least one of the following models: diffusion model, generative adversarial network, variational autoencoder, and stream-based generative model.

[0019] In one embodiment, when the spatial constraint information is a binary mask and the preset generation model is a diffusion model, the spatial constraint information is used as a conditional input to control the preset generation model to synthesize sample content that conforms to the corresponding difficulty category features within a specified area of ​​the background image, including: Obtain the original binary mask; The original binary mask is dilated to obtain the dilated binary mask. In the initial stage of denoising in the diffusion model, the dilated binary mask is used as spatial constraint information to control the diffusion model to synthesize sample content within a specified area of ​​the background image. In the denoising termination stage of the diffusion model, the original binary mask is used as spatial constraint information to control the diffusion model to synthesize sample content within a specified area of ​​the background image.

[0020] In one embodiment, synthetic samples are fused with real training samples, and the parameters of the target detection model are updated using the fused samples, including: In each training mini-batch, samples are taken from the real training sample set and the synthetic sample set according to a preset participation ratio; The sampled real training samples and synthetic samples are concatenated to form the current training mini-batch; Based on the current training mini-batch, determine the joint loss to update the parameters of the object detection model.

[0021] In one embodiment, determining the joint loss includes: Based on the subset of real training samples in the current training mini-batch, determine the detection loss for the subset of real training samples; Determine the detection loss for the synthetic sample subset based on the synthetic sample subset in the current training mini-batch; The joint loss is determined by the weighted sum of the detection loss of the real training sample subset and the detection loss of the synthetic sample subset; where the weight of each loss is determined based on the proportion of the corresponding sample in the current training mini-batch.

[0022] In one embodiment, updating the parameters of the target detection model includes: Freeze the backbone network parameters or feature extraction layer parameters of the target detection model, and update the detection head parameters or classification layer parameters of the target detection model. Unfreeze all parameters of the object detection model and update the parameters of the entire object detection model at a learning rate lower than that of the previous update process.

[0023] In one embodiment, it further includes: At the end of each training round, obtain the validation score of the object detection model on the validation dataset; When the validation score is better than the historical best validation score, update the model checkpoint with the target detection model of the current training round. Parameter updates will stop when the verification score does not exceed the historical best verification score for a consecutive preset number of rounds.

[0024] In one embodiment, the preset category is a fabric defect category, and the target detection model is a fabric defect detection model.

[0025] Secondly, this application also provides a model optimization system, including: The acquisition module is used to obtain performance feedback information of the object detection model on the validation dataset. The diagnostic module is used to determine the learning difficulty of the target detection model in each preset category based on performance feedback information, and to determine at least one difficult category from each preset category based on the learning difficulty. The quota allocation module is used to determine the amount of synthetic data corresponding to each difficulty category based on the learning difficulty of each difficulty category; The synthesis module is used to generate synthetic samples for each difficulty category based on the amount of synthetic data. The synthetic samples carry annotation information. The optimization module is used to fuse synthetic samples with real training samples and use the fused samples to update the parameters of the target detection model.

[0026] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0027] The model optimization method, system, and electronic device provided in this application acquire performance feedback information of the object detection model on a validation dataset. This performance feedback information accurately reflects the actual learning effect of the object detection model on each preset category in the current state. Then, based on this performance feedback information, the learning difficulty of the object detection model on each preset category can be determined, and at least one difficult category can be identified from each preset category based on the learning difficulty. This allows for precise localization of the detector's learning bottleneck, enabling subsequent augmentation processing to focus on the target categories that the detector truly struggles to master. Furthermore, the amount of synthetic data corresponding to each difficult category can be determined based on the learning difficulty of each difficult category, ensuring that the scale of the synthetic data matches the actual difficulty level of each difficult category. Categories with higher learning difficulty can obtain more synthetic samples, thus enabling on-demand allocation of synthetic resources. By generating synthetic samples carrying labeled information for each difficult category based on the amount of synthetic data, and using the fusion of synthetic samples with real training samples to update the parameters of the object detection model, the object detection model can obtain targeted supplementary supervision for its own performance bottleneck during the parameter update process. Thus, this application can achieve precise coupling between the parameter update process of the object detection model and its actual learning bottleneck. Without the need for manual pre-setting of enhancement strategies, the model can autonomously identify and specifically compensate for its learning deficiencies in various categories, thereby improving the recognition accuracy and overall detection performance of the object detection model in difficult categories. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart illustrating a model optimization method provided in an embodiment of this application; Figure 2 A schematic diagram of a process for determining learning difficulty is provided for an embodiment of this application; Figure 3 This application provides a schematic diagram of a process for determining the amount of synthesized data in an embodiment. Figure 4 A schematic diagram illustrating a process for generating annotation information for sample content, provided in an embodiment of this application; Figure 5 This application provides a schematic diagram of the distribution of samples of each category in a dataset. Figure 6A schematic diagram of a normalized diagnostic confusion matrix provided in an embodiment of this application; Figure 7 A schematic diagram showing the comparison between a synthetic sample and a real reference sample provided in an embodiment of this application; Figure 8 A schematic diagram comparing the detection results and attention heatmaps of a baseline model and an optimized model provided in an embodiment of this application; Figure 9 A structural block diagram of a model optimization system provided in an embodiment of this application; Figure 10 This is a schematic diagram of the internal structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] To alleviate the problem of insufficient training data, existing methods all share a common fundamental flaw: the generative augmentation process is independent of the target detector's learning state, forming an open-loop architecture. Specifically, whether it's a defect synthesis method based on generative adversarial networks or an augmentation strategy based on diffusion models, the sample generation process does not rely on the target detector's performance feedback on real data. Fixed allocation strategies (such as uniform allocation or equal allocation based on sample frequency) or random generation strategies only determine the synthesis quantity for each defect category according to preset rules, failing to identify the target detector's true learning difficulty in each category (e.g., a category with a high false negative rate due to complex textures, or severe inter-class confusion due to visual similarity of defects). This leads to two problems: firstly, synthesis resources are evenly or blindly allocated to categories that the detector has mastered well, resulting in wasted computational resources; secondly, difficult categories where the detector's performance is insufficient cannot receive targeted supplementary supervision that matches their actual difficulty level. Meanwhile, the generated synthetic samples require additional manual annotation before they can be used for the re-optimization training of the detector, which further increases the implementation cost. Moreover, when synthetic samples are simply mixed into the training set, there is a lack of a refined fusion strategy that matches the performance bottleneck of the detector, making it difficult to form a continuous improvement loop in real iterative development scenarios.

[0032] In one exemplary embodiment, Figure 1 This is a flowchart illustrating a model optimization method provided in an embodiment of this application, as shown below. Figure 1As shown, a model optimization method is provided. Taking the application of this method to a server as an example, it can be understood that this method can also be applied to terminals, and to systems including terminals and servers, and implemented through the interaction between the terminal and the server. Terminals can be, but are not limited to, various personal computers, laptops, smartphones, and tablets. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services. In this embodiment, the method includes the following steps S101 to S105. Wherein: S101. Obtain performance feedback information of the object detection model on the validation dataset.

[0033] Here, the object detection model refers to a pre-trained machine learning model used to identify and locate target objects in images. The validation dataset refers to a set of samples separated from real-world labeled samples that are not used for training the object detection model's parameters but are only used to evaluate model performance and tune hyperparameters. Performance feedback information refers to information reflecting the object detection model's detection performance across various categories on the validation dataset.

[0034] For example, the prediction results of the object detection model on a validation dataset can be obtained as performance feedback information. The prediction results include the object detection model's category predictions and / or location detection information for each sample in the validation dataset. By obtaining this performance feedback information, one can understand the object detection model's detection performance on the validation dataset.

[0035] As an example, we can obtain the object detection model's category prediction results and bounding box prediction results for each object on the validation dataset, and statistically analyze the detection performance for each category.

[0036] By obtaining performance feedback information of the object detection model on the validation dataset, the actual learning effect of the object detection model on each preset category in the current state can be reflected without interfering with the model training process, providing a data foundation for improving the targeting of subsequent parameter updates.

[0037] S102. Based on the performance feedback information, determine the learning difficulty of the target detection model in each preset category, and based on the learning difficulty, determine at least one difficult category from each preset category.

[0038] Here, "pre-defined categories" refers to categories pre-divided based on the morphological characteristics or causal types of the target object. "Learning difficulty" is a quantitative indicator used to characterize the difficulty of the object detection model in learning a particular pre-defined category; this indicator is determined based on the performance feedback information of the object detection model on the validation dataset. "Difficult categories" refers to one or more pre-defined categories selected from all pre-defined categories that have a high learning difficulty; these are the key targets for subsequent data augmentation.

[0039] For example, the server system can statistically analyze the false detection information of each preset category based on performance feedback information, and then determine the learning difficulty of each preset category based on the false detection information. After determining the learning difficulty of each preset category, the preset categories are sorted from high to low according to their learning difficulty, and the preset categories with the highest preset number of preset categories are determined as difficult categories; or, the preset categories with learning difficulty exceeding a preset threshold are determined as difficult categories.

[0040] As an example, the server system can obtain the prediction results of the object detection model on a validation dataset, calculate the false negative rate and / or inter-class confusion rate for each preset category, determine the total error rate based on the false negative rate, and determine the inter-class confusion penalty term based on the inter-class confusion rate. Based on the weighted sum of the total error rate and the inter-class confusion penalty term for each preset category, the learning difficulty of each preset category is determined. Optionally, the weight coefficient of the inter-class confusion penalty term can be configured according to the actual application scenario to adjust the influence of inter-class confusion on the evaluation of learning difficulty. Subsequently, the preset categories are sorted from high to low learning difficulty, and a preset number of the top-ranked preset categories are selected as the difficult categories.

[0041] By determining the learning difficulty of each preset category based on performance feedback information and selecting the difficult categories accordingly, subsequent data augmentation processing can focus on target categories that are truly difficult for the target detection model to master, thereby improving the targeting of parameter updates.

[0042] S103. Determine the amount of synthetic data corresponding to each difficulty category based on the learning difficulty level of each difficulty category.

[0043] The amount of synthetic data refers to the number of synthetic samples that need to be generated for a particular difficulty category.

[0044] For example, the server system can obtain the number of real samples for each difficulty category in the real training set, determine the frequency compensation amount for each difficulty category based on the number of real samples, determine the difficulty compensation amount for each difficulty category based on the learning difficulty, and then determine the amount of synthetic data for each difficulty category based on the weighted sum of the frequency compensation amount and the difficulty compensation amount.

[0045] As an example, the frequency compensation amount for each difficulty category is determined based on the difference between the preset reference number and the actual sample number. When the actual sample number is greater than or equal to the preset reference number, the frequency compensation amount is zero. The difficulty compensation amount for each difficulty category is determined based on the product of the learning difficulty and the preset reference number. The synthetic data volume for each difficulty category is determined based on the weighted sum of the frequency compensation amount and the difficulty compensation amount. Optionally, the synthetic data volume can be determined according to the following formula:

[0046] in, This indicates the amount of synthetic data for category c. This indicates the preset reference quantity. This represents the number of true samples in category c. This represents the normalized learning difficulty of category c. Indicates the difficulty compensation intensity coefficient. This represents the set of difficulty categories.

[0047] Optional, The number of synthetic defect instances in category c can be rounded to the nearest integer.

[0048] By determining the amount of synthetic data corresponding to each difficulty category based on the learning difficulty, the scale of synthetic data can be matched with the actual difficulty level of each difficulty category. Categories with higher learning difficulty will receive more synthetic samples, thereby enabling on-demand allocation of synthetic resources.

[0049] S104. Based on the amount of synthetic data, generate synthetic samples for each difficulty category. The synthetic samples carry annotation information.

[0050] Here, synthetic samples refer to image data containing the target object generated by a generative model. Annotation information refers to the category labels and / or location annotations corresponding to the synthetic samples.

[0051] For example, the server system can obtain the background image and the corresponding spatial constraint information, use the spatial constraint information as a conditional input, control the preset generation model to synthesize sample content that conforms to the corresponding difficulty category features in a specified area of ​​the background image, and generate annotation information of the sample content according to the spatial constraint information.

[0052] As an example, a pre-defined generative model can be used to obtain a background image and its corresponding binary mask. The binary mask is then input into the generative model as a spatial constraint, controlling the model to synthesize the target object within the masked region of the background image. After synthesis, connected component analysis is performed on the binary mask to determine the minimum bounding rectangle of each connected component. Based on these minimum bounding rectangles, bounding box annotations for each target object are automatically generated. Optionally, the spatial constraint information can also be a semantic segmentation map, keypoint coordinates, or bounding box coordinates. When the spatial constraint information is bounding box coordinates, the annotation information can be directly determined based on the bounding box coordinates. Optionally, the pre-defined generative model can be a diffusion model, a generative adversarial network, a variational autoencoder, or a flow-based generative model.

[0053] By utilizing spatial constraint information to guide the generative model to synthesize target objects within a specified region, the generation location and shape of the synthesized samples can be precisely controlled. At the same time, by automatically generating annotation information using the spatial constraint information used in the generation process, the synthesized samples can be directly used for supervised training of the target detection model without additional manual annotation, thereby lowering the threshold for using synthesized samples.

[0054] S105. Fuse the synthetic samples with the real training samples, and use the fused samples to update the parameters of the target detection model.

[0055] In this context, fusion refers to combining synthetic samples and their corresponding annotations with real training samples to form the training data source used for updating model parameters. The real training sample set refers to the collection of all images with real annotations and their annotation information in the original training dataset. Parameter update refers to the process of further optimizing the model parameters using the fused training data, based on the initial training of the object detection model.

[0056] For example, the server system can add the synthetic samples and their annotation information to the training data pool, and together with the original real training samples, form a fused training set. The fused training set can then be used to further update the parameters of the object detection model.

[0057] As an example, in each training mini-batch, samples are taken from both the real training sample set and the synthetic sample set according to a preset participation ratio. The sampled real training samples and synthetic samples are concatenated to form the current training mini-batch. The joint loss is calculated based on the current training mini-batch, and the model parameters are updated. Optionally, the preset participation ratio is less than 0.5, ensuring that real samples remain the majority in each training mini-batch. Optionally, during parameter updates, the backbone network parameters of the object detection model are first frozen, and the detection head parameters are updated; then all parameters are unfrozen, and the entire object detection model is fine-tuned with a learning rate lower than the previous update process. Optionally, at the end of each training epoch, the validation score of the object detection model on the validation dataset is obtained; when the validation score is better than the historical best validation score, the model checkpoint is updated with the object detection model of the current training epoch; when the validation score does not exceed the historical best validation score for a consecutive preset epoch, parameter updates are stopped.

[0058] By fusing synthetic samples with real training samples to update the parameters of the object detection model, the object detection model can obtain targeted supplementary supervision to address its performance bottlenecks during the parameter update process. Without compromising the learning effect on the real data distribution, it can make up for the object detection model's insufficient learning in difficult categories, thereby improving the recognition accuracy and overall detection performance of the object detection model in difficult categories.

[0059] In this embodiment, by acquiring performance feedback information of the object detection model on the validation dataset, which reflects the actual learning effect of the object detection model on each preset category in the current state, the learning difficulty of the object detection model on each preset category is determined based on this performance feedback information. At least one difficult category is then identified from each preset category based on the learning difficulty, thereby pinpointing the learning bottleneck of the object detection model. Furthermore, the amount of synthetic data corresponding to each difficult category is determined based on the learning difficulty of each difficult category, ensuring that the scale of the synthetic data matches the actual difficulty level of each category. Synthetic samples carrying labeled information are generated for each difficult category based on the amount of synthetic data. These synthetic samples are then fused with real training samples, and the fused samples are used to update the parameters of the object detection model. This allows the object detection model to receive targeted supplementary supervision to address its own performance bottlenecks during the parameter update process. Thus, the parameter update process of the object detection model is precisely coupled with its actual learning bottlenecks, enabling the object detection model to autonomously identify and specifically compensate for its learning deficiencies in each category, thereby improving the recognition accuracy and overall detection performance of the object detection model in difficult categories.

[0060] In one exemplary embodiment, performance feedback information includes the prediction results of the object detection model on the validation dataset; Figure 2A flowchart illustrating the process of determining learning difficulty is provided in this embodiment of the application, such as... Figure 2 As shown, in step S102, based on performance feedback information, the learning difficulty of the object detection model in each preset category is determined, including: S201. Based on the prediction results, statistical analysis is performed on the false detection information for each preset category, including the false detection rate and / or the inter-class confusion rate.

[0061] S202. Determine the learning difficulty of each preset category based on the false negative rate and / or inter-class confusion rate.

[0062] Among these, performance feedback information refers to information reflecting the detection performance of the object detection model on the validation dataset for each category. Prediction results refer to the category prediction information and / or location detection information output by the object detection model after detecting each object on the validation dataset. False detection information refers to statistical information about object detection models that failed to correctly identify objects during the detection process. The false negative rate refers to the proportion of object objects in a certain preset category that are missed (i.e., not detected) by the object detection model; a higher false negative rate indicates a greater degree to which the object detection model "cannot find" that category of objects. The inter-class confusion rate refers to the proportion of object objects in a certain preset category that are misidentified as objects of other preset categories by the object detection model; a higher inter-class confusion rate indicates a greater degree to which the object detection model "cannot distinguish" that category of objects from other categories. Learning difficulty refers to a quantitative indicator used to characterize the difficulty of the object detection model in learning a certain preset category.

[0063] For example, the server system can compare the prediction results of the object detection model on the validation dataset with the true annotation information of each sample in the validation dataset, and count the number of correctly detected, missed, and misdetected objects of each preset category, thereby obtaining the false negative rate and / or inter-class confusion rate of each preset category. Subsequently, based on the false negative rate and / or inter-class confusion rate, the learning difficulty of each preset category is determined. Specifically, a higher false negative rate indicates that the object detection model is less sensitive to the detection response of that category of objects, and the learning difficulty is higher; a higher inter-class confusion rate indicates that the objects of that category are more similar in features to objects of other categories, resulting in insufficient discriminative ability of the object detection model, and the learning difficulty is higher.

[0064] As an example, the false negative rate (FRR) for each preset category can be calculated as the proportion of target objects predicted as the background category out of the total number of actual target objects in that category. The inter-class confusion rate (ICR) can be calculated as the sum of the proportions of target objects predicted as other preset categories out of the total number of actual target objects in that category. Then, the FRR and ICR are multiplied by their respective weighting coefficients and summed to obtain the learning difficulty for that category; alternatively, the sum of the FRR and ICR can be used as the learning difficulty for that category. Optionally, the FRR and ICR can be normalized separately before determining the learning difficulty for each preset category to eliminate the impact of an imbalance in the number of target objects between different categories on the difficulty evaluation.

[0065] In this embodiment, by statistically analyzing the false negative rate and / or inter-class confusion rate of each preset category based on the prediction results, and determining the learning difficulty of each preset category accordingly, the learning difficulty of the target detection model in each preset category can be comprehensively evaluated from two dimensions: false negative (cannot be found) and confusion (cannot be distinguished). This ensures that the determined learning difficulty can truly reflect the actual performance bottleneck of the target detection model, thereby improving the accuracy of subsequent location of difficult categories.

[0066] In an exemplary embodiment, step S202, determining the learning difficulty of each preset category based on the false negative rate and / or inter-class confusion rate, may specifically include: The total error rate is determined based on the false negative rate, and the inter-class confusion penalty is determined based on the inter-class confusion rate. The learning difficulty of each preset category is determined by the weighted sum of the total error rate of each preset category and the inter-class confusion penalty.

[0067] The total error rate refers to the proportion of target objects in a certain preset category that are not correctly detected by the target detection model, i.e., the false negative rate. The inter-class confusion penalty term is a weighting term applied to the inter-class confusion rate, used to adjust the contribution of inter-class confusion to the evaluation of learning difficulty. The weighted sum is the combined value obtained by multiplying the total error rate and the inter-class confusion penalty term by their respective weighting coefficients, and this value serves as the learning difficulty for that preset category.

[0068] For example, the server system can determine the total error rate of each preset category based on the false negative rate of each preset category, and determine the inter-class confusion penalty term of each preset category based on the inter-class confusion rate of each preset category. Subsequently, the total error rate of each preset category and the inter-class confusion penalty term are weighted and summed to obtain the learning difficulty of each preset category.

[0069] In this embodiment, by determining the total error rate based on the false negative rate and the inter-class confusion penalty based on the inter-class confusion rate, and then determining the learning difficulty of each preset category based on the weighted sum of the total error rate and the inter-class confusion penalty, the false detection information of two different dimensions, false negative and confusion, can be uniformly incorporated into the evaluation system of learning difficulty. This allows the learning difficulty to reflect both the degree to which the target detection model "cannot find" the target of that category and the degree to which the target detection model "cannot distinguish" the target of that category from other target categories, thereby improving the comprehensiveness and accuracy of the learning difficulty in representing the actual learning bottleneck of the target detection model.

[0070] In one exemplary embodiment, the learning difficulty is determined according to the following formula:

[0071] in, This represents the learning difficulty of category c. This represents the correct detection rate for category c. This represents the overall error rate (i.e., the rate of missed detections). This represents the inter-class confusion rate where the true class c is predicted as class j. denoted by , where C represents the predefined set of categories.

[0072] Specifically, total error rate It can also be decomposed into the sum of the false negative rate and the inter-class confusion rate, that is:

[0073] in, This represents the false negative rate when the true class c is predicted as the background class. This represents the sum of inter-class confusion rates when the true class c is predicted as any of the other predefined classes. Therefore, the learning difficulty... The first item The second term represents the total error rate for category c. Apply additional penalties to inter-class confusion.

[0074] Optionally, the inter-class confusion penalty coefficient α can be in the range of 0 ≤ α ≤ 1. When α = 0, the learning difficulty degenerates into the total error rate. Increasing α will make the category with higher inter-class confusion rate obtain higher learning difficulty.

[0075] In an exemplary embodiment, after calculating the learning difficulty of each preset category, the learning difficulty of each preset category can be normalized. Normalization refers to mapping the learning difficulty of each preset category to a uniform numerical range to facilitate subsequent selection of difficulty categories and allocation of synthetic data.

[0076] For example, the server system obtains the minimum and maximum values ​​of the learning difficulty of all preset categories, and maps the learning difficulty of each preset category to the numerical range of [0,1] by dividing the difference between the learning difficulty of each preset category and the minimum value by the difference between the maximum value and the minimum value.

[0077] As an example, normalized learning difficulty Determine according to the following formula:

[0078] in, This represents the minimum learning difficulty among all preset categories. This represents the maximum learning difficulty across all preset categories, where ε is a small constant to prevent division by zero. Optionally, when When the learning difficulty of all preset categories is equal, normalization may not be performed, or the normalized learning difficulty of all categories may be set to the same value. Optionally, other normalization methods may be used, such as mapping the learning difficulty to the interval [0,1].

[0079] By normalizing the learning difficulty of each preset category, the differences in the numerical range of learning difficulty between different categories can be eliminated, allowing the learning difficulty of each preset category to be compared and ranked on a uniform scale. This is beneficial for the subsequent selection of difficulty categories and the allocation of synthetic data.

[0080] In one exemplary embodiment, at least one difficulty category is determined from a set of preset categories based on learning difficulty, including: Sort the preset categories according to their learning difficulty from highest to lowest; The categories with the highest sorting counts are designated as the difficult categories; Alternatively, a preset category whose learning difficulty exceeds a preset threshold can be defined as a difficulty category.

[0081] Here, "learning difficulty" refers to a quantitative indicator used to characterize the degree of difficulty a target detection model faces in learning a specific pre-defined category. "Pre-defined category" refers to categories pre-divided based on the morphological characteristics or causal type of the target object. "Difficult category" refers to one or more pre-defined categories with high learning difficulty selected from all pre-defined categories; these are the focus of subsequent data augmentation. "Pre-defined quantity" refers to the pre-set number of difficult categories selected. "Pre-defined threshold" refers to a pre-defined learning difficulty threshold value used to determine whether a given pre-defined category belongs to the difficult category.

[0082] For example, after determining the learning difficulty of each preset category, the server system sorts the preset categories in descending order of learning difficulty. After sorting, a number of the top-ranked preset categories can be selected as difficult categories based on a preset quantity; alternatively, preset categories whose learning difficulty exceeds a preset threshold can be identified as difficult categories. The difficult categories selected in this way are those for which the object detection model performs poorly in the current state and require targeted data augmentation.

[0083] As an example, the preset quantity H can be set to 5, meaning that the preset categories are sorted from highest to lowest learning difficulty, and the top 5 preset categories are identified as the difficult categories. Specifically, the set of difficult categories... Determined in the following manner:

[0084] That is, selecting the learning difficulty from the entire preset category set C. The top H categories are designated as the difficult categories. Optionally, the preset number H can be configured according to the needs of the actual application scenario. For example, H can be set to 3, 5, 8, or 10 to adapt to the requirements of the number of difficult categories in different scenarios.

[0085] Optionally, the specific value of the preset threshold can be configured according to the needs of the actual application scenario. When the learning difficulty of a certain preset category exceeds the preset threshold, the preset category is determined as a difficult category.

[0086] Optionally, the difficulty category can be determined by combining two methods: selecting the top-ranked preset number and exceeding the threshold. For example, among the categories whose learning difficulty exceeds the preset threshold, a preset number of categories with the highest ranking can be further selected as the difficulty category.

[0087] In this embodiment, by sorting each preset category according to the learning difficulty from high to low and determining the top preset number of categories as difficult categories, or by determining the difficult categories by the learning difficulty exceeding a preset threshold, the preset categories that are most difficult to learn can be flexibly selected based on the actual performance feedback of the object detection model. This allows subsequent targeted enhancement to focus on the real performance bottleneck of the object detection model, thereby improving the targeting and efficiency of parameter updates.

[0088] In one exemplary embodiment, Figure 3 This application provides a schematic diagram of a process for determining the amount of synthesized data in an embodiment, as shown below. Figure 3 As shown, in step S103, determining the amount of synthetic data corresponding to each difficulty category includes: S301. Obtain the number of real samples for each difficulty category in the real training set; S302. Determine the frequency compensation amount for each difficulty category based on the actual number of samples. S303. Determine the difficulty compensation amount for each difficulty category based on the degree of learning difficulty; S304. Determine the amount of composite data for each difficulty category based on the weighted sum of the frequency compensation amount and the difficulty compensation amount.

[0089] In this context, the real training set refers to the set of labeled samples used to train the object detection model. The number of real samples refers to the number of real samples included in the real training set for a given category. The difficult category refers to one or more predefined categories selected from all predefined categories that have a higher learning difficulty. The frequency compensation is a synthetic quantity component determined based on the number of real samples, used to compensate for the insufficient sample frequency of the difficult category. The difficulty compensation is a synthetic quantity component determined based on the learning difficulty, used to add an additional synthetic quota for categories with higher learning difficulty. The weighted sum is the combined value obtained by multiplying the frequency compensation and difficulty compensation by their respective weight coefficients and then adding them together. The synthetic data volume refers to the number of synthetic samples that need to be generated for a given difficult category.

[0090] For example, for each preset category labeled as a difficult category, the server system obtains the actual number of samples for that category, determines a frequency compensation amount based on the difference between the actual number of samples and a preset benchmark, and simultaneously determines a difficulty compensation amount based on the learning difficulty of that category. The frequency compensation amount and the difficulty compensation amount are then weighted and summed to obtain the synthetic data volume corresponding to that difficult category. In this way, the synthetic data volume can both compensate for the objective deficiency in the number of samples for difficult categories and respond to their specific learning difficulty requirements.

[0091] In this embodiment, the frequency compensation amount is determined by obtaining the number of real samples for each difficulty category in the real training set, and the difficulty compensation amount is determined based on the learning difficulty. Then, the amount of synthetic data for each difficulty category is determined by the weighted sum of the frequency compensation amount and the difficulty compensation amount. This ensures that the determination of the amount of synthetic data takes into account both the objective insufficiency of the difficulty category in terms of the number of samples and the subjective bottleneck in terms of the model learning difficulty. The more scarce the real samples and the higher the learning difficulty of the category, the more synthetic quota it receives. This enables the on-demand allocation of synthetic resources among different difficulty categories and improves the utilization efficiency of synthetic data.

[0092] In one exemplary embodiment, determining the amount of synthetic data for each difficulty category includes: The frequency compensation amount for each difficulty category is determined based on the difference between the preset reference number and the actual sample number; where the frequency compensation amount is zero when the actual sample number is greater than or equal to the preset reference number. The difficulty compensation amount for each difficulty category is determined based on the product of the learning difficulty and the preset reference quantity. The amount of synthetic data for each difficulty category is determined by the weighted sum of the frequency compensation and the difficulty compensation for each difficulty category.

[0093] The following are the key parameters for calculating the difficulty of a target detection model: **Preset Reference Quantity:** This refers to a pre-defined reference quantity based on the training requirements of the target detection model or the sample distribution, used to measure the sufficiency of samples for each difficulty category. **True Sample Quantity:** This refers to the number of true samples included in the actual training set for a given difficulty category. **Frequency Compensation:** This is a synthetic quantity component determined based on the difference between the preset reference quantity and the true sample quantity, used to compensate for frequency gaps when the number of true samples is insufficient. **Learning Difficulty:** This is a quantitative indicator representing the difficulty of learning a target detection model for a given category. **Difficulty Compensation:** This is a synthetic quantity component determined based on the product of the learning difficulty and the preset reference quantity, used to add additional synthetic quotas for categories with higher learning difficulty. **Weighted Sum:** This is the combined value obtained by multiplying the frequency compensation and difficulty compensation by their respective weighting coefficients and then summing them. **Synthetic Data Quantity:** This refers to the number of synthetic samples that need to be generated for a given difficulty category.

[0094] For example, for each preset category labeled as a difficult category, the server system obtains the number of actual samples for that category, calculates the difference between the preset reference number and the number of actual samples, and uses this difference as a frequency compensation amount when the number of actual samples is less than the preset reference number; when the number of actual samples is greater than or equal to the preset reference number, the frequency compensation amount is zero. Simultaneously, the server system multiplies the learning difficulty of that difficult category by the preset reference number to obtain a difficulty compensation amount. Then, the frequency compensation amount and the difficulty compensation amount are weighted and summed to obtain the amount of synthetic data corresponding to that difficult category.

[0095] Optionally, the preset reference quantity can be determined based on the number of real samples in the head category (i.e., one or more categories with a large number of real samples). For example, it can be the average or minimum number of real samples in each category of the head category. Optionally, when the number of real samples has reached or exceeded the preset reference quantity, the frequency compensation amount is zero, that is, no synthetic quota is allocated due to insufficient sample quantity, but a synthetic quota can still be obtained through the difficulty compensation amount due to high learning difficulty. Optionally, upper limits can also be set for the frequency compensation amount and the difficulty compensation amount respectively to avoid generating an excessive number of synthetic samples for a certain category.

[0096] In this embodiment, the frequency compensation amount is determined based on the difference between the preset reference number and the actual sample number. The frequency compensation amount is limited to zero when the actual sample number is greater than or equal to the preset reference number. This ensures that frequency compensation only applies to difficult categories with insufficient sample numbers, avoiding the allocation of synthetic quotas based on frequency factors when the sample number is sufficient. Simultaneously, the difficulty compensation amount is determined based on the product of the learning difficulty and the preset reference number, ensuring that categories with higher learning difficulty receive more difficulty compensation. Finally, the final synthetic data volume is determined by a weighted sum of the frequency compensation amount and the difficulty compensation amount. This achieves a combined determination of the synthetic quota from two dimensions: "sufficient sample number" and "difficulty in model learning," thereby improving the rationality and accuracy of the synthetic data volume determination. Ultimately, this enhances the recognition accuracy and overall detection performance of the target detection model in difficult categories.

[0097] In one exemplary embodiment, the amount of synthesized data is determined according to the following formula:

[0098] in, This indicates the amount of synthetic data for category c. Indicates the preset reference quantity. This represents the number of true samples in category c. This represents the normalized learning difficulty of category c. Indicates the difficulty compensation intensity coefficient. This represents the set of difficulty categories.

[0099] In one exemplary embodiment, generating synthetic samples for each difficulty category based on the amount of synthetic data includes: Obtain the background image and its corresponding spatial constraint information; Spatial constraint information is used as a conditional input to control the preset generation model to synthesize sample content that conforms to the corresponding difficulty category features within a specified area of ​​the background image; Based on spatial constraint information, generate annotation information for the sample content.

[0100] Here, the background image refers to the original image that does not contain the target object. Its image type corresponds to the difficulty category to be synthesized. For example, in a fabric defect detection scenario, the background image could be a fabric texture image without defects. Spatial constraint information refers to information used to limit the position, shape, or extent of the target object in the synthesized sample in image space. This information provides spatial guidance for the generative model to synthesize the region. The preset generative model refers to a pre-set machine learning model that can generate image content based on input conditions. The specified region refers to a specific region in the background image defined by the spatial constraint information, within which the generative model synthesizes sample content. Sample content refers to the target object synthesized within the specified region of the background image that conforms to the characteristics of the corresponding difficulty category. Labeling information refers to the category label and / or location labeling information corresponding to the synthesized sample.

[0101] For example, the server system obtains a corresponding number of background images and their corresponding spatial constraint information based on the amount of synthetic data for each difficulty category. The spatial constraint information is then used as input to a preset generation model, controlling the model to synthesize sample content that conforms to the characteristics of the corresponding difficulty category within a specified area of ​​the background images. After synthesis, annotation information for the sample content is automatically generated based on the spatial constraint information used in the generation process.

[0102] As an example, a pre-defined generative model (such as a diffusion model) can be used. For each difficult category to be synthesized, the background image and corresponding binary mask of that category are obtained. The binary mask is input into the diffusion model as a spatial constraint, controlling the diffusion model to synthesize sample content that conforms to the characteristics of the corresponding difficult category within the mask region of the background image. After synthesis, connected component analysis is performed on the binary mask to determine the minimum bounding rectangle of each connected component. Based on each minimum bounding rectangle, the bounding box annotation information of each sample content is automatically generated. Optionally, the spatial constraint information can also be a semantic segmentation map, keypoint coordinates, or bounding box coordinates. When the spatial constraint information is bounding box coordinates, the annotation information can be directly determined based on the bounding box coordinates without performing connected component analysis. Optionally, the pre-defined generative model can be a diffusion model, a generative adversarial network, a variational autoencoder, or a flow-based generative model.

[0103] In this embodiment, by acquiring the background image and its corresponding spatial constraint information, and using the spatial constraint information as a conditional input to control the preset generation model to synthesize sample content that conforms to the characteristics of the corresponding difficulty category within a specified area, the directional generation of synthesized samples is realized, making the spatial position of the synthesized samples controlled by the preset spatial constraint information. At the same time, by automatically generating annotation information for the sample content based on the spatial constraint information, the synthesized samples can be directly used for supervised training of the object detection model without additional manual annotation, which can reduce the threshold for using synthesized samples and the implementation cost. This can efficiently provide supplementary training data with annotation information for difficult categories, which is beneficial to improving the recognition accuracy and overall detection performance of the object detection model in difficult categories.

[0104] In one exemplary embodiment, the spatial constraint information includes at least one of the following: a binary mask, a semantic segmentation map, key point coordinates, and bounding box coordinates; Figure 4 This is a schematic diagram illustrating a process for generating annotation information for sample content, as provided in an embodiment of this application. Figure 4 As shown, based on spatial constraint information, annotation information for the sample content is generated, including: S401. Perform connected component analysis on the spatial constraint information to determine the boundary information of each connected component; S402. Generate corresponding bounding box annotations based on the boundary information of each connected component; Alternatively, S403, determine the corresponding bounding box annotation based on the spatial constraint information.

[0105] Spatial constraint information refers to information used to limit the position, shape, or range of the target object in the image space within the synthesized sample. A binary mask is a binary matrix of the same size as the background image, where the pixel region corresponding to the first value represents the synthesized region of the target object, and the pixel region corresponding to the second value represents the background-preserved region. A semantic segmentation map is an image that assigns a category label to each pixel in the image, where different categories correspond to different pixel values ​​or color identifiers, used to distinguish different regions or target objects in the image. Keypoint coordinates refer to the coordinate information of points in the image used to represent the key positions of the target object. Bounding box coordinates refer to the coordinate information of the target object's position represented by a rectangular box in the image, usually represented by the coordinates of the top-left and bottom-right corners of the rectangle. Connected component analysis refers to performing connectivity analysis on adjacent pixel regions with the same pixel value in the image, classifying connected pixel regions into the same connected component and disconnected regions into different connected components. Boundary information refers to information used to define the range of each connected component in the image space. Bounding box annotation refers to the annotation information representing the target object's position in the form of a rectangular box, usually represented by the coordinates of the top-left and bottom-right corners of the rectangle.

[0106] For example, after acquiring spatial constraint information, the server system can generate annotation information for the sample content in an appropriate manner based on the type of spatial constraint information. When the spatial constraint information is a binary mask or a semantic segmentation map, connected component analysis can be performed on the spatial constraint information to divide the unconnected pixel regions into different connected components, determine the boundary information of each connected component in the image space, and then generate corresponding bounding box annotations based on the boundary information of each connected component. When the spatial constraint information is bounding box coordinates, the bounding box coordinates can be directly used as annotation information. When the spatial constraint information is keypoint coordinates, the corresponding bounding box annotations can be determined based on the keypoint coordinates. For example, a bounding box of a predetermined size can be generated centered on the keypoint coordinates, or the smallest bounding rectangle enclosing all keypoints can be determined as the bounding box based on multiple keypoint coordinates.

[0107] As an example, for spatial constraint information in the form of a binary mask, the binary mask can be decomposed into connected components to obtain the pixel support domains of each connected component. For each pixel support domain of a connected component, its minimum x-coordinate, minimum y-coordinate, maximum x-coordinate, and maximum y-coordinate in the image coordinate system are determined. The minimum x-coordinate and minimum y-coordinate determine the upper left corner of the rectangle, and the maximum x-coordinate and maximum y-coordinate determine the lower right corner of the rectangle, forming the minimum bounding rectangle enclosing the pixel support domain of the connected component. This minimum bounding rectangle serves as the bounding box label for the sample content corresponding to the connected component. Optionally, when the spatial constraint information is a semantic segmentation map, connected component analysis can be performed on the pixel regions belonging to the target category in the semantic segmentation map, and bounding box labels for each connected component can be generated in a manner similar to that of the binary mask. Optionally, when the spatial constraint information is bounding box coordinates, these bounding box coordinates can be directly used as the bounding box label for the sample content without performing connected component analysis. Optionally, when the spatial constraint information is the coordinates of key points, the corresponding bounding box annotation can be generated based on the coordinates of key points, for example, generating a bounding box with the key point coordinates as the center and a preset size.

[0108] In this embodiment, spatial constraint information, including at least one of binary masks, semantic segmentation maps, keypoint coordinates, or bounding box coordinates, provides a flexible spatial constraint method for generating synthetic samples. Simultaneously, by performing connected component analysis on the spatial constraint information to determine the boundary information of each connected component and generating corresponding bounding box annotations, or by directly determining the corresponding bounding box annotations based on the spatial constraint information, automatic generation of sample content annotation information can be achieved. This allows different types of spatial constraint information to be adapted to generate corresponding annotation information, eliminating the need for manual intervention in the annotation process, lowering the barrier to entry for using synthetic samples, and efficiently providing supplementary training data with accurate annotation information for difficult categories. This is beneficial for improving the recognition accuracy and overall detection performance of the target detection model in difficult categories.

[0109] In one exemplary embodiment, the preset generative model includes at least one of the following models: diffusion model, generative adversarial network, variational autoencoder, and stream-based generative model.

[0110] Among these, a pre-defined generative model refers to a machine learning model that is pre-set and capable of generating image content based on input conditions. A diffusion model is a generative model that generates images from random noise through a progressive denoising process. A generative adversarial network (GAN) is a generative model that generates images through adversarial training between a generator and a discriminator. A variational autoencoder (VAE) is a generative model that maps input data to a latent space using an encoder and then reconstructs the data from the latent space using a decoder. A stream-based generative model is a generative model that maps a simple distribution to a complex data distribution through a series of invertible transformations.

[0111] For example, the server system can select one of the following as a preset generative model based on actual application requirements: a diffusion model, a generative adversarial network, a variational autoencoder, or a stream-based generative model. Spatial constraint information is then used as a conditional input to the selected preset generative model, which is then used to synthesize sample content conforming to the characteristics of the corresponding difficulty category within a specified region of the background image. Different generative models have different generative characteristics and applicable scenarios, and can be selected and configured according to actual needs.

[0112] As an example, a diffusion model can be used as the preset generative model. The diffusion model generates images from random noise through progressive denoising, exhibiting strong image synthesis capabilities and relatively stable training performance. The background image and its corresponding spatial constraint information (e.g., a binary mask) are used as conditional inputs to the diffusion model, controlling it to synthesize sample content conforming to the features of the corresponding hard category within the masked region of the background image, while the background region outside the mask remains unchanged. Optionally, when using a generative adversarial network as the preset generative model, spatial constraint information and random noise can be jointly input into the generator to synthesize sample content within the region defined by the spatial constraint information. Optionally, when using a variational autoencoder as the preset generative model, spatial constraint information and latent variables can be jointly input into the decoder to reconstruct or synthesize sample content within the region defined by the spatial constraint information. Optionally, when using a flow-based generative model as the preset generative model, a conditional flow model can be used to generate sample content within the region defined by the spatial constraint information.

[0113] In this embodiment, by pre-setting a generation model including at least one of diffusion model, generative adversarial network, variational autoencoder or flow-based generation model, multiple optional model architectures are provided for the generation of synthetic samples. This enables the method of this application to adapt to different application scenarios and generation requirements, thereby improving the flexibility and applicability of the method. It is also beneficial to generate diverse supplementary training data for difficult categories, thereby improving the recognition accuracy and overall detection performance of the target detection model in difficult categories.

[0114] In an exemplary embodiment, when the spatial constraint information is a binary mask and the preset generation model is a diffusion model, the spatial constraint information is used as a conditional input to control the preset generation model to synthesize sample content that conforms to the corresponding difficulty category features within a specified region of the background image, including: Obtain the original binary mask; The original binary mask is dilated to obtain the dilated binary mask. In the initial stage of denoising in the diffusion model, the dilated binary mask is used as spatial constraint information to control the diffusion model to synthesize sample content within a specified area of ​​the background image. In the denoising termination stage of the diffusion model, the original binary mask is used as spatial constraint information to control the diffusion model to synthesize sample content within a specified area of ​​the background image.

[0115] Spatial constraint information refers to information used to limit the position, shape, or range of the target object in the image space within the synthesized sample. A binary mask is a binary matrix of the same size as the background image, where the pixel region corresponding to the first value represents the synthesized region of the target object, and the pixel region corresponding to the second value represents the background retention region. The original binary mask refers to the initial binary mask without any image processing operations. Dilation refers to an image morphological operation that expands the pixel region corresponding to the first value in the binary mask, extending the boundary of that region outward by a certain pixel distance. The dilated binary mask refers to the binary mask obtained after dilation, where the region corresponding to the first value is larger than that of the original binary mask. A diffusion model refers to a generative model that generates an image from random noise through a stepwise denoising process. The initial denoising stage refers to the early steps of the diffusion model that gradually denoise from random noise. The final denoising stage refers to the later steps of the diffusion model that are close to completing denoising and about to output the final synthesized image.

[0116] For example, when a diffusion model is used as the preset generation model and the spatial constraint information is a binary mask, the server system obtains the original binary mask and dilates it to obtain an dilated binary mask. In the initial stage of denoising using the diffusion model, the dilated binary mask is used as the spatial constraint information. At this time, the boundary of the spatial constraint region is relatively loose, allowing the generated content to gradually form over a larger area. In the final stage of denoising using the diffusion model, the system switches back to the original binary mask as the spatial constraint information. At this time, the spatial constraint region shrinks to its original size, ensuring precise alignment between the boundary of the generated content and the original mask. By gradually transitioning the mask from an dilated state to its original state during the denoising process, progressive control of the synthesis region boundary from loose to precise can be achieved.

[0117] As an example, an original binary mask can be obtained. A dilation operation from image morphology is then used to expand the region corresponding to the first value in the original binary mask by a preset radius (e.g., 8 pixels), resulting in a dilated binary mask. At the initial time step of the diffusion model's denoising process, the dilated binary mask is input as spatial constraint information into the diffusion model, controlling the model to synthesize sample content within the dilated region. As the denoising time steps progress, the dilation radius gradually decreases. At the final time step of the denoising process, the original binary mask is input as spatial constraint information into the diffusion model, controlling the model to complete the final synthesis of sample content within the original mask region. Optionally, the dilation radius can be gradually reduced from 8 pixels to 0 pixels, and the reduction can be linear, exponential, or according to a preset decay curve. Optionally, the initial value and variation curve of the dilation radius can be configured and adjusted according to the target size of the synthesized sample, the complexity of the background texture, and the generation quality requirements. Optionally, before dilating the original binary mask, it can also be smoothed to eliminate jagged edges or discontinuities at the mask boundaries.

[0118] In this embodiment, by using an expanded binary mask as spatial constraint information in the initial stage of denoising of the diffusion model and the original binary mask as spatial constraint information in the final stage of denoising, the synthesis region gradually shrinks from loose to precise during the synthesis process. This is beneficial for the synthesized content to obtain more sufficient generation space and more natural texture transition in the initial stage of denoising, and to accurately fit the boundary of the original mask in the final stage of denoising. This can improve the visual fusion between the target object and the background in the synthesized sample, making the synthesized sample more realistic and natural. In turn, it is beneficial to improve the recognition accuracy and overall detection performance of the target detection model after updating parameters using the synthesized sample.

[0119] In an exemplary embodiment, synthetic samples are fused with real training samples, and the parameters of the target detection model are updated using the fused samples, including: In each training mini-batch, samples are taken from the real training sample set and the synthetic sample set according to a preset participation ratio; The sampled real training samples and synthetic samples are concatenated to form the current training mini-batch; Based on the current training mini-batch, determine the joint loss to update the parameters of the object detection model.

[0120] In this dataset, the real training sample set refers to the collection of all images with real annotations and their annotation information in the original training dataset. The synthetic sample set refers to the collection of synthetic samples with annotation information generated by a pre-defined generative model. The training mini-batch refers to the subset of samples used for each parameter update during the object detection model's parameter update process. The pre-defined participation ratio refers to the target proportion of synthetic samples to the total number of samples in each training mini-batch. Concatenation refers to combining the subsets of samples obtained from the real training sample set and the synthetic sample set into a complete training mini-batch. The joint loss refers to the total loss calculated on the training mini-batch, used to drive the parameter updates of the object detection model.

[0121] For example, during the construction of each training mini-batch, the server system independently samples from the real training sample set and the synthetic sample set according to a preset participation ratio. The sampled real sample subset and synthetic sample subset are then concatenated to form the current training mini-batch. The joint loss on this training mini-batch is calculated, and the parameters of the object detection model are updated based on the joint loss using backpropagation and an optimizer.

[0122] As an example, if the size of the training mini-batch is set to m, and the preset participation ratio is γ, then the target number of synthetic samples is . The target number of real samples is The actual proportion of synthesized samples is In each training step, samples are randomly taken from the real training sample set. One sample was randomly selected from the synthetic sample set. The system takes a sample and concatenates the two samples to form the current training mini-batch. It then calculates the joint loss based on the current training mini-batch and updates the parameters of the object detection model. Optionally, the preset participation ratio γ can be configured to be less than 0.5, ensuring that the number of real samples in each training mini-batch is greater than the number of synthetic samples. This allows real samples to dominate the training process and avoids the potential distribution bias of synthetic samples from dominating the training process.

[0123] Optionally, the actual proportion of synthesized samples ρ can be equal to or approximately equal to the preset participation proportion γ, when After rounding and When there are minor deviations, the difference between the actual proportion ρ and γ is within an acceptable range.

[0124] Optionally, the preset participation ratio can remain fixed during training, or it can gradually change with the increase of training rounds. Optionally, the joint loss can include the detection loss on a subset of real samples and the detection loss on a subset of synthetic samples.

[0125] In this embodiment, by sampling from both the real training sample set and the synthetic sample set according to a preset participation ratio in each training mini-batch, and concatenating the sampled real training samples and synthetic samples into the current training mini-batch, each training mini-batch contains both real and synthetic samples. This allows the target detection model to learn from both the real and synthetic data distributions simultaneously during parameter updates. Furthermore, by controlling the proportion of synthetic samples in each training mini-batch through the preset participation ratio, the potential distribution bias of synthetic samples can be prevented from dominating the training process. Finally, by determining the joint loss based on the current training mini-batch to update the parameters of the target detection model, synthetic samples can be effectively integrated into the training process as supplementary supervisory signals. This allows for targeted compensation of the target detection model's insufficient learning in difficult categories without compromising the learning effect on the real data distribution, thereby improving the recognition accuracy and overall detection performance of the target detection model in difficult categories.

[0126] In one exemplary embodiment, determining the joint loss includes: Based on the subset of real training samples in the current training mini-batch, determine the detection loss for the subset of real training samples; Determine the detection loss for the synthetic sample subset based on the synthetic sample subset in the current training mini-batch; The joint loss is determined by a weighted sum of the detection loss of a subset of real training samples and the detection loss of a subset of synthetic samples; where the weight of each loss is determined based on the proportion of the corresponding samples in the current training mini-batch.

[0127] In this context, the current training mini-batch refers to the subset of samples used in the current round of parameter updates for the object detection model. The real training sample subset refers to the set of samples sampled from the real training sample set in the current training mini-batch. The synthetic sample subset refers to the set of samples sampled from the synthetic sample set in the current training mini-batch. Detection loss is a measure of the difference between the object detection model's predictions on the sample subset and the ground truth labels, used to evaluate the current detection performance of the object detection model. Joint loss is the total loss calculated on the training mini-batch, obtained by weighted summation of the detection losses from the real training sample subset and the synthetic sample subset, used to drive parameter updates for the object detection model. Weighted summation is the combined value obtained by multiplying the detection losses from the real training sample subset and the synthetic sample subset by their respective weights and then summing them. Quantity proportion refers to the proportion of samples from the real training sample subset or synthetic sample subset in the current training mini-batch to the total number of samples in the training mini-batch.

[0128] For example, the server system calculates the detection loss for the subset of real training samples and the detection loss for the subset of synthetic samples in the current training mini-batch, respectively. Then, it determines the weight of each loss based on the proportion of real and synthetic samples in the current training mini-batch, and weights the two detection losses according to their respective proportions to obtain the joint loss. Subsequently, it uses this joint loss to calculate the gradient and backpropagates it to update the parameters of the object detection model.

[0129] As an example, let's assume that the current training mini-batch is formed by concatenating a subset of real training samples and a subset of synthetic samples, where the actual proportion of synthetic samples is ρ, and the actual proportion of real samples is (1-ρ). The joint loss is determined according to the following formula:

[0130] in, Indicates joint loss, This represents the detection loss of a subset of real training samples. This represents the detection loss of the synthetic sample subset. This indicates the actual proportion of synthetic samples in the current training mini-batch. This indicates the actual percentage of the real sample.

[0131] Specifically, detection loss can include classification loss and bounding box localization loss.

[0132] Optionally, the weights of each loss can be non-uniformly configured according to actual needs. For example, a higher weight can be assigned to the detection loss of the real training sample subset to emphasize the dominant role of real data, or a higher weight can be assigned to the detection loss of the synthetic sample subset to enhance the supplementary supervision effect on difficult categories. Optionally, before calculating the joint loss, the detection loss of the real training sample subset and the detection loss of the synthetic sample subset can be normalized separately to eliminate the influence of the different numerical ranges of the two detection losses on the weighting result.

[0133] In this embodiment, the detection loss of the real training sample subset is determined based on the real training sample subset in the current training mini-batch, and the detection loss of the synthetic sample subset is determined based on the synthetic sample subset. Then, the joint loss is determined based on the weighted sum of the two according to the corresponding sample quantity ratio. This ensures that the contributions of real samples and synthetic samples in the joint loss match their respective quantity proportions in the current training mini-batch. As a result, during the parameter update process, the target detection model can learn from both the real data distribution and the synthetic data distribution simultaneously. At the same time, it ensures that the loss contribution of real samples is dominant and prevents the potential distribution bias of synthetic samples from dominating the training process. This is beneficial to improving the recognition accuracy and overall detection performance of the target detection model in difficult categories.

[0134] In an exemplary embodiment, updating the parameters of the target detection model includes: Freeze the backbone network parameters or feature extraction layer parameters of the target detection model, and update the detection head parameters or classification layer parameters of the target detection model. Unfreeze all parameters of the object detection model and update the parameters of the entire object detection model at a learning rate lower than that of the previous update process.

[0135] In this context, the object detection model refers to a pre-trained machine learning model used to identify and locate target objects in images. Backbone network parameters refer to the parameters of the backbone network in the object detection model responsible for extracting general image features from the input image. Feature extraction layer parameters refer to the parameters of the feature extraction layer in the object detection model responsible for extracting features from the input image; their function corresponds to the backbone network parameters. Detection head parameters refer to the parameters of the detection head network in the object detection model responsible for classifying the target category and regressing bounding boxes based on the extracted features. Classification layer parameters refer to the parameters of the classification layer in the object detection model responsible for classifying the target category; their function corresponds to the detection head parameters. Freezing refers to keeping some parameters unchanged during parameter updates, preventing them from participating in gradient updates. The learning rate is a hyperparameter that controls the magnitude of parameter adjustment during parameter updates; a lower learning rate results in smaller parameter adjustments and more refined parameter updates.

[0136] For example, when updating the parameters of an object detection model using fused samples, a two-stage progressive update strategy can be adopted. In the first stage, the backbone network parameters or feature extraction layer parameters of the object detection model are frozen, remaining unchanged during the parameter update process; only the detection head parameters or classification layer parameters are updated. In the second stage, all parameters of the object detection model are unfrozen, and all parameters of the entire object detection model are updated using a lower learning rate than in the first stage. Through this two-stage progressive update, the model first quickly adapts to the distribution characteristics of the synthetic data without destroying the learned general feature representations, and then the overall model is fine-tuned to achieve global optimization.

[0137] As an example, in the first stage of parameter update, all parameters of the backbone network of the object detection model are frozen, and only the parameters of the detection head are updated. A first learning rate is used for parameter update, and the model is trained for a predetermined number of epochs to allow the detection head to quickly adapt to the distribution characteristics of the synthetic data. In the second stage of parameter update, all parameters of the backbone network are unfrozen, and a second learning rate lower than the first learning rate is used to fine-tune all parameters of the entire object detection model. This is trained for a predetermined number of epochs to achieve global optimization. Optionally, the ratio of the first learning rate to the second learning rate can be 10:1, 5:1, or 2:1, etc., and the specific value can be adjusted according to the actual training effect. Optionally, in the first stage, a layer-by-layer unfreezing method can also be used, that is, gradually unfreezing some backbone network layers within the first stage, so that the scope of parameter update gradually expands from the detection head to the backbone network, thereby achieving a smoother training transition. Optionally, if the parameter descriptions of the feature extraction layer and classification layer are used, the backbone network parameters correspond to the feature extraction layer parameters, and the detection head parameters correspond to the classification layer parameters.

[0138] In this embodiment, a two-stage progressive update strategy is implemented by first freezing the backbone network parameters or feature extraction layer parameters of the object detection model and updating the detection head parameters or classification layer parameters, and then unfreezing all parameters and updating the entire object detection model with a lower learning rate. In the first stage, freezing the backbone network parameters and updating only the detection head parameters allows the detection head to quickly adapt to the distribution characteristics of synthetic data without destroying the general feature representations already learned by the backbone network. In the second stage, fine-tuning the entire object detection model with a lower learning rate allows for global fine-tuning without destroying the learned general features, thus avoiding the feature forgetting problem that may be caused by direct full network updates. This is beneficial for improving the recognition accuracy and overall detection performance of the object detection model in difficult categories.

[0139] In one exemplary embodiment, it further includes: At the end of each training round, obtain the validation score of the object detection model on the validation dataset; When the validation score is better than the historical best validation score, update the model checkpoint with the target detection model of the current training round. Parameter updates will stop when the verification score does not exceed the historical best verification score for a consecutive preset number of rounds.

[0140] In this context, "training epochs" refers to the process of a complete forward and backward propagation of the object detection model across the entire training set during training. "Validation dataset" refers to a set of samples partitioned from real-world labeled samples that are not used for training the object detection model's parameters but are solely used to evaluate model performance and tune hyperparameters. "Validation score" is a quantitative metric calculated on the validation dataset to measure the model's current performance. "Best historical validation score" refers to the highest validation score achieved by the object detection model on the validation dataset during the current training process. "Model checkpoint" refers to a snapshot of the model parameters saved during training, used to select the best-performing model version after training. "Preset epochs" refers to a pre-defined number of training epochs used to determine whether the validation score has failed to improve over a sufficiently long training period, thus deciding whether to stop parameter updates.

[0141] For example, at the end of each training epoch, the server system evaluates the object detection model obtained in the current training epoch on the validation dataset and calculates its validation score. The validation score of the current epoch is compared with the historical best validation score. If the validation score of the current epoch is better than the historical best validation score, the model checkpoints are updated using the object detection model of the current training epoch. If the validation score of the current epoch fails to exceed the historical best validation score, the number of epochs without improvement is accumulated. Parameter updates are stopped when the validation score fails to exceed the historical best validation score for a preset number of consecutive epochs.

[0142] As an example, at the end of each training epoch for parameter updates in the object detection model, the server system obtains the validation score of the object detection model on the validation dataset and compares the current epoch's validation score with the historical best validation score. If the current epoch's validation score is higher than the historical best validation score, the model checkpoint is updated based on the object detection model of the current training epoch, and the number of epochs without improvement is reset to zero. If the current epoch's validation score does not exceed the historical best validation score, the number of epochs without improvement is incremented by one. When the number of epochs without improvement reaches a preset number of epochs (e.g., 10 epochs), parameter updates are stopped, and the model checkpoint corresponding to the historical best validation score is restored as the final parameter update result. Optionally, the validation score can be any combination of one or more evaluation metrics such as mean precision, recall, accuracy, or F1 score. Optionally, the preset number of epochs can be configured according to the size of the validation dataset, the total number of training epochs, or actual application requirements; for example, the preset number of epochs can be set to 5 epochs, 10 epochs, 20 epochs, etc. Optionally, after stopping parameter updates, you can return to the steps described above: obtaining performance feedback information, determining the learning difficulty, determining the difficulty category, determining the amount of synthetic data, generating synthetic samples, and performing fusion training. The updated target detection model can then be used as the new target detection model for the next round of iterative optimization until the overall performance meets the preset requirements.

[0143] In this embodiment, by obtaining the validation score at the end of each training round, updating the model checkpoint when the validation score is better than the historical best validation score, and stopping parameter updates when the validation score does not exceed the historical best validation score for a preset number of consecutive rounds, an early stopping mechanism and a model selection mechanism based on the performance feedback of the validation dataset are implemented. This mechanism ensures that the selection of model checkpoints is always based on the performance of the object detection model on real data rather than on synthetic data, ensuring that the model optimization direction is always guided by real data performance. This avoids the potential bias of synthetic data from negatively impacting model selection, thereby helping to prevent the model from overfitting to the synthetic data distribution. At the same time, it can effectively save training time and computing resources, which is beneficial to improving the recognition accuracy and overall detection performance of the object detection model after parameter updates in difficult categories.

[0144] In one exemplary embodiment, the preset category is a fabric defect category, and the target detection model is a fabric defect detection model.

[0145] Here, "preset category" refers to a category pre-defined based on the morphological characteristics or cause type of the target object. "Fabric defect category" refers to a defect classification pre-defined based on the morphological characteristics, texture background, or cause type of fabric surface defects. "Target detection model" refers to a pre-trained machine learning model used to identify and locate target objects from images. "Fabric defect detection model" refers to a pre-trained machine learning model used to identify and locate defects from fabric images.

[0146] For example, when the preset category is a fabric defect category and the target detection model is a fabric defect detection model, the preset category can be a defect category pre-divided according to the morphological features or texture background of the fabric defect, such as defect types under different texture backgrounds; the target detection model is a fabric defect detection model used to detect and locate various types of fabric defects from fabric images. The method provided in this application can be applied to fabric defect detection scenarios. By obtaining the performance feedback information of the fabric defect detection model on the validation dataset, the learning difficulty of the fabric defect detection model on each preset defect category is determined, and at least one difficult defect category is determined from each preset defect category based on the learning difficulty. The corresponding amount of synthetic data is determined according to the learning difficulty of each difficult defect category, and synthetic defect samples for each difficult defect category are generated. After fusing the synthetic defect samples with the real training samples, the parameters of the fabric defect detection model are updated, thereby improving the detection performance of the fabric defect detection model on difficult defect categories.

[0147] As an example, the preset categories can be a predefined set of defect categories in the fabric defect detection scenario, such as defect categories under different texture backgrounds, including houndstooth textures, bamboo textures, and solid color textures. The fabric defect detection model can be a deep learning model for detecting and locating defects in fabric images. In actual industrial scenarios, the number of defect samples under different texture backgrounds often exhibits a long-tail distribution. Defect samples under certain complex textures (such as houndstooth and bamboo textures) are relatively scarce, and defect features are easily masked by high-frequency background signals, resulting in poor detection performance of the fabric defect detection model on these categories. The method provided in this application can, based on the performance feedback information of the fabric defect detection model on the validation dataset, filter out difficult defect categories with insufficient detection performance, and achieve targeted optimization of the fabric defect detection model by generating synthetic defect samples of the corresponding difficult categories and fusing them for training. Optionally, the preset categories can also be defect categories in other industrial visual inspection scenarios, such as defect categories in electronic component surface defect detection scenarios or automotive part surface defect detection scenarios. Optionally, the target detection model can also be a target detection model applied to autonomous driving scenarios, a target detection model applied to security monitoring scenarios, etc.

[0148] In this embodiment, by pre-setting the category as the fabric defect category and the target detection model as the fabric defect detection model, the method provided in this application can be specifically applied to the fabric defect detection scenario. It performs targeted enhancement on the performance bottleneck of the fabric defect detection model on different fabric texture defect categories, enabling the fabric defect detection model to autonomously identify and specifically compensate for its learning deficiencies in various fabric defects. Thus, without the need for manually pre-setting enhancement strategies, the recognition accuracy and overall detection performance of the fabric defect detection model on difficult defect categories are improved.

[0149] In some exemplary embodiments, the normalized diagnostic confusion matrix described above can be constructed as follows: For each preset category in the validation dataset, the number of defective instances under each real category that are predicted to belong to each category (including the background category) is counted, and normalization is performed using the total number of instances in each real category as the denominator to obtain the normalized diagnostic confusion matrix.

[0150] Specifically, set This represents the number of defect instances of true category c that are assigned to predicted category j, where, Unmatched real instances are assigned to `bg` and considered missed detections. The normalized confusion statistic is defined as:

[0151] Here, the first subscript represents the predicted category, and the second subscript represents the true category. For each true category c, Indicates the correct detection ratio. Indicates the percentage of missed detections. time This indicates confusion between defect categories. The total error rate can be decomposed into the sum of the missed detection rate and the inter-class confusion rate:

[0152] In some exemplary embodiments, after calculating the learning difficulty of each preset category, the preset categories are sorted from highest to lowest learning difficulty, and a preset number of categories at the top of the sorting are determined as the difficult categories. Specifically, the set of difficult categories is determined in the following manner:

[0153] That is, selecting the learning difficulty from the entire preset category set C. The top 5 categories are designated as the hard categories. The selected hard categories are fixed before targeted synthesis and final test set evaluation to ensure a consistent evaluation benchmark throughout the parameter update process.

[0154] In some exemplary embodiments, such as Figure 5As shown, the long-tailed distribution of defect instances in the train-core subset is the long-tailed distribution of defect samples within the train-core subset; Figure 5 The horizontal axis represents the defect category, and the vertical axis corresponds to the number of defect instances in the train-core subset. On the horizontal axis, Pattern2 represents pattern 2; Knot Pattern represents knotted patterns (fabric surface defects); Brown Plaid represents brown plaid fabric; Blue Plaid represents blue plaid fabric; Twill Plaid represents twill fabric; Gingham represents checkered fabric (yarn-dyed small checkered fabric); Pattern4 represents pattern 4; Houndstooth represents houndstooth fabric; Red Plaid represents red plaid fabric; Floral Print1 represents floral print 1 (Floral Print: printed fabric); Floral Print3 represents floral print 3; Floral Print2 represents floral print 2; Gray Plaid represents gray plaid fabric; Pattern1 represents pattern 1; Thick Stripe represents thick stripes; Thin Stripe represents thin stripes; Dot Pattern represents polka dot patterns; White Plain represents white plain fabric (Plain = plain fabric); Pattern3 represents pattern 3.

[0155] Depend on Figure 5 As can be seen, the number of samples varies significantly across different defect categories. The red boxes mark the top 5 LDI-hard categories with the largest number of samples; the dashed boxes mark the tail-25% (5 categories) of low-frequency defect categories. The overall dataset exhibits a typical long-tail distribution: the head defect categories have a sufficient number of samples, while the tail defect categories have significantly fewer samples, with the number of samples decreasing gradually from the head categories to the tail categories. This shows that the training subset of the ZJU-Leaper dataset exhibits a clear long-tail distribution across different defect categories, and the low-frequency categories are not entirely consistent with the LDI-hard categories, indicating that relying solely on sample frequency is insufficient to guide targeted augmentation.

[0156] like Figure 6 As shown, Figure 6 The horizontal axis represents the true class of the defective sample, and the vertical axis represents the predicted class output by the model. The color bar on the right represents the normalized proportion; the darker the color, the higher the normalized proportion. The values ​​in the diagonal squares of the matrix represent the normalized proportion of each class correctly predicted. The "background" row at the bottom of the matrix corresponds to the normalized proportion of each class that was mispredicted as background. Among them, WhitePlain: white plain weave fabric; Thick Stripe: thick stripe; Thin Stripe: thin stripe; Dot Pattern: polka dot pattern; Houndstooth: houndstooth; Gingham: checkered fabric; Knot Pattern: knotted pattern; Twill Plaid: twill plaid fabric; Blue Plaid: blue plaid fabric; Brown Plaid: brown plaid fabric; Gray Plaid: gray plaid fabric; Red Plaid: red plaid fabric; Floral Print 1: floral print 1; Floral Print 2: floral print 2; Floral Print 3: floral print 3; Pattern 1: pattern 1; Pattern 2: pattern 2; Pattern 3: pattern 3; Pattern 4: pattern 4; background: background (defect-free background area); This confusion matrix visually illustrates the model's classification performance for various defect categories: diagonal elements represent the percentage of correct identification for each defect category, off-diagonal elements represent misclassification between categories, and the bottom background rows reflect the proportion of each defect category that is misclassified as background areas. It can be used to evaluate the model's identification and false negative rates for head categories and low-frequency tail defect categories. In the normalized diagnostic confusion matrix of the baseline detector on the validation set, the diagonal elements range from 0.80 to 0.97, the off-diagonal elements between defect categories are negligible, and the remaining errors are mainly concentrated in the background rows (false negatives), with a false negative rate ranging from 0.03 to 0.20.

[0157] like Figure 7 As shown, Figure 7 Each row in the image corresponds to a different defect category, and each column consists of the background image (Ibg), the master mask image (Mmask), the composite image of the superimposed bounding box (Bbox) (Isyn+Bbox), and the real reference sample (RealReference).

[0158] The first column, Ibg, represents the original fabric background image; the second column, Mmask, represents the master mask image of the defect region, with white areas representing the defect mask region and black areas representing the fabric background; the third column, Isyn+Bbox, represents the defect image synthesized based on the mask, with green rectangles marking the defect bounding boxes detected by the model; the fourth column, Real Reference, represents the corresponding real reference sample, with red rectangles marking the actual location of the real defect. Among them, Pattern2: Pattern 2; KnotPattern: Knot pattern; Gray Plaid: Gray plaid fabric; Dot Pattern: Dot pattern; Floral Print1: Floral print 1; Floral Print3: Floral print 3. This schematic diagram visually compares the synthesized defect sample with the real defect sample, showing that the morphology and location distribution of the defects synthesized by this scheme have a high similarity to real fabric defects. The green bounding boxes can effectively locate the synthesized defect region, verifying the effectiveness of the defect synthesis scheme and its potential for expanding samples for long-tail defect datasets.

[0159] The synthesized defect samples are spatially constrained by a predefined mask, exhibiting category-related variations in shape, scale, and local texture. The synthesized region can achieve visual fusion in different fabric backgrounds such as repeating textures, checks, dots, and floral patterns, and the automatically generated bounding box covers the positive mask area.

[0160] like Figure 8 As shown, Figure 8 Rows (a), (b), (c), (d), and (e) correspond to different defect samples of white plain fabric, floral print 2, floral print 3, pattern 3, and knot pattern, respectively. The four columns are, in order, the baseline prediction result, the baseline attention map, the LDI-GA prediction result, and the LDI-GA attention map.

[0161] In the prediction results image, different colored boxes mark the defect detection boxes output by the model and their corresponding defect category labels; in the attention heatmap, the warmer the color, the higher the attention weight the model assigns to that image region. Figure 8This is a comparison chart of the model prediction results and attention visualization, with four columns for horizontal pairwise comparison: Column 1: Defect detection prediction chart of the Baseline model, with colored boxes marking defects detected by the model; Column 2: Attention heatmap corresponding to the Baseline model, with warm-colored areas representing image regions that the model focuses on; Column 3: Defect detection prediction chart of this application passed LDI-GA; Column 4: Attention heatmap corresponding to LDI-GA.

[0162] The comparison reveals that the baseline model suffers from defect omissions; the LDI-GA model accurately detects defects missed by the baseline model, and its high-response region of the attention heatmap more precisely focuses on the actual location of the defect. This indicates that the LDI-GA method can guide the model to pay more attention to the effective features of the defect, reduce interference from irrelevant background areas, and improve the performance of fabric defect detection. LDI-GA is the target detection model proposed in this application.

[0163] The LDI-GA-optimized detector exhibits a more concentrated activation response around the marked defect area. For low-contrast defects, defects embedded in dense floral textures, and defects appearing in highly repetitive textures, the response map shows more concentrated activation and higher spatial correspondence.

[0164] In some exemplary implementations, the overall process of the LDI-GA framework includes three stages: performance diagnostics, targeted synthesis, and model re-optimization. First, validation set confusion statistics are used to calculate category-level LDI scores and determine category-specific synthesis quotas. Then, mask-constrained AnomalyDiffusion generates targeted defect instances and their automatically generated bounding box annotations. Finally, through dual-path sampling and two-stage progressive fine-tuning, real and synthetic samples are incorporated into the detector re-optimization, with model checkpoints selected solely based on the real validation set.

[0165] In some exemplary embodiments, the two-stage detector re-optimization process described above can be executed according to the following algorithm flow. In the initialization phase, the detector parameters are decomposed into backbone network parameters. and detection head parameters Set the participation factor for synthetic samples And calculate the target number of synthetic samples in each mini-batch. The target number of real samples is The actual proportion of synthesized samples is In the first phase, the backbone network parameters were frozen. Only update the detection head parameters Using learning rate In the second stage, all parameters are unfrozen, and a learning rate lower than that in the first stage is adopted. Fine-tune the entire detector. At the end of each training epoch, calculate the validation score on the validation dataset. Update the model checkpoint when the validation score is better than the historical best validation score. If the validation score does not improve for a preset number of consecutive epochs, terminate the current stage of training and restore the best checkpoint.

[0166] In some exemplary implementations, when using DSFR-DETR as the object detection model and ZJU-Leaper as the dataset, the specific parameter configuration of the above method can be as follows: inter-class confusion penalty coefficient α = 0.5, difficulty compensation strength coefficient β = 0.5, synthetic sample participation factor γ = 0.4, and preset reference number. The number of instances is determined based on the head category. Under this configuration, the total synthesis quota for the first 5 difficult categories (Pattern2, Thick Stripe, Knot Pattern, FloralPrint3, White Plain) is 2,621 synthesized defective instances. In each training mini-batch, the actual proportion of synthesized samples (ρ) is approximately 0.4, and the proportion of real samples is approximately 0.6, ensuring that real samples remain the majority in each mini-batch. The joint loss is calculated according to... Confirmed, among which This represents the detection loss of a subset of real training samples. This represents the detection loss of the synthetic sample subset.

[0167] In some exemplary embodiments, the preset categories in the above method can be fabric defect categories, and the target detection model can be a fabric defect detection model. The ZJU-Leaper dataset contains 19 texture categories, of which 4 pattern categories correspond to solid color fabric textures, containing a total of 71,127 defect-free images and 27,650 defective images. The 63,814 images in the original training set were stratified and sampled at a 9:1 ratio, further divided into a core training subset and a validation subset, while the original test set remained unchanged. The core training subset is used for detector training and synthetic sample generation, while the validation subset is used only for constructing the confusion matrix, calculating LDI, selecting hyperparameters, and selecting checkpoints. The test set is strictly isolated from detector training, synthetic guidance, and parameter tuning.

[0168] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0169] The model optimization system provided in the embodiments of this application is described below. The model optimization system has the same inventive concept as the model optimization method described above. The solution to the problem provided by the system is similar to the solution described in the method described above. Therefore, the specific limitations of one or more model optimization system embodiments provided below can be referred to the limitations of the model optimization method above. The model optimization system described below and the model optimization method described above can be referred to each other, and will not be repeated here.

[0170] In one exemplary embodiment, Figure 9 This is a schematic diagram of the structure of a model optimization system provided in an embodiment of this application, as shown below. Figure 9 As shown, the model optimization system 90 includes: Module 910 is used to obtain performance feedback information of the object detection model on the validation dataset. The diagnostic module 920 is used to determine the learning difficulty of the target detection model in each preset category based on the performance feedback information, and to determine at least one difficult category from each preset category based on the learning difficulty. The quota allocation module 930 is used to determine the amount of synthetic data corresponding to each difficulty category based on the learning difficulty of each difficulty category; The synthesis module 940 is used to generate synthetic samples for each difficulty category based on the amount of synthetic data. The synthetic samples carry annotation information. The optimization module 950 is used to fuse synthetic samples with real training samples and use the fused samples to update the parameters of the target detection model.

[0171] In an exemplary embodiment, the performance feedback information includes the prediction results of the object detection model on the validation dataset; the diagnostic module 920 can be specifically used to: statistically analyze the false detection information of each preset category based on the prediction results, wherein the false detection information includes the false negative rate and / or the inter-class confusion rate; and determine the learning difficulty of each preset category based on the false negative rate and / or the inter-class confusion rate.

[0172] The diagnostic module 920 can be used to: determine the total error rate based on the missed detection rate and determine the inter-class confusion penalty based on the inter-class confusion rate; and determine the learning difficulty of each preset category based on the weighted sum of the total error rate and the inter-class confusion penalty for each preset category.

[0173] In one exemplary embodiment, the learning difficulty is determined according to the following formula:

[0174] in, This represents the learning difficulty of category c. This represents the correct detection rate for category c. This represents the total error rate. This represents the inter-class confusion rate where the true class c is predicted as class j. denoted by , where C represents the predefined set of categories.

[0175] In an exemplary embodiment, the diagnostic module 920 is specifically configured to: sort the preset categories according to their learning difficulty from high to low; and determine the preset categories with the highest number of sorted categories as the difficult categories; Alternatively, a preset category whose learning difficulty exceeds a preset threshold can be defined as a difficulty category.

[0176] In an exemplary embodiment, the quota allocation module 930 is specifically used to: obtain the number of real samples for each difficulty category in the real training set; determine the frequency compensation amount for each difficulty category based on the number of real samples; determine the difficulty compensation amount for each difficulty category based on the learning difficulty; and determine the amount of synthetic data for each difficulty category based on the weighted sum of the frequency compensation amount and the difficulty compensation amount.

[0177] In an exemplary embodiment, the quota allocation module 930 is specifically used to: determine the frequency compensation amount for each difficulty category based on the difference between the preset reference quantity and the actual sample quantity; wherein, when the actual sample quantity is greater than or equal to the preset reference quantity, the frequency compensation amount is zero; determine the difficulty compensation amount for each difficulty category based on the product of the learning difficulty and the preset reference quantity; and determine the synthetic data amount for each difficulty category based on the weighted sum of the frequency compensation amount and the difficulty compensation amount for each difficulty category.

[0178] In one exemplary embodiment, the amount of synthesized data is determined according to the following formula:

[0179] in, This indicates the amount of synthetic data for category c. Indicates the preset reference quantity. This represents the number of true samples in category c. This represents the normalized learning difficulty of category c. Indicates the difficulty compensation intensity coefficient. This represents the set of difficulty categories.

[0180] In an exemplary embodiment, the synthesis module 940 is specifically used to: acquire a background image and corresponding spatial constraint information; use the spatial constraint information as a condition input to control a preset generation model to synthesize sample content that conforms to the characteristics of the corresponding difficulty category within a specified area of ​​the background image; and generate annotation information for the sample content based on the spatial constraint information.

[0181] In an exemplary embodiment, the spatial constraint information includes at least one of the following: a binary mask, a semantic segmentation map, key point coordinates, and bounding box coordinates; the synthesis module 940 is specifically used to: perform connected component analysis on the spatial constraint information to determine the boundary information of each connected component; generate corresponding bounding box annotations based on the boundary information of each connected component; or, determine the corresponding bounding box annotations based on the spatial constraint information.

[0182] In one exemplary embodiment, the preset generative model includes at least one of the following models: diffusion model, generative adversarial network, variational autoencoder, and stream-based generative model.

[0183] In an exemplary embodiment, when the spatial constraint information is a binary mask and the preset generation model is a diffusion model, the synthesis module 940 is specifically used to: obtain the original binary mask; perform dilation processing on the original binary mask to obtain a dilated binary mask; in the initial stage of denoising the diffusion model, use the dilated binary mask as spatial constraint information to control the diffusion model to synthesize sample content in a specified area of ​​the background image; in the final stage of denoising the diffusion model, use the original binary mask as spatial constraint information to control the diffusion model to synthesize sample content in a specified area of ​​the background image.

[0184] In an exemplary embodiment, the optimization module 950 is specifically configured to: sample from the real training sample set and the synthetic sample set according to a preset participation ratio in each training mini-batch; concatenate the sampled real training samples and synthetic samples to form the current training mini-batch; and determine the joint loss based on the current training mini-batch to update the parameters of the target detection model.

[0185] In an exemplary embodiment, the optimization module 950 is specifically configured to: determine the detection loss of the real training sample subset based on the real training sample subset in the current training mini-batch; determine the detection loss of the synthetic sample subset based on the synthetic sample subset in the current training mini-batch; and determine the joint loss based on the weighted sum of the detection loss of the real training sample subset and the detection loss of the synthetic sample subset; wherein the weight of each loss is determined based on the proportion of the corresponding sample quantity in the current training mini-batch.

[0186] In an exemplary embodiment, the optimization module 950 is specifically used to: freeze the backbone network parameters or feature extraction layer parameters of the target detection model, update the detection head parameters or classification layer parameters of the target detection model; unfreeze all parameters of the target detection model, and update the parameters of the entire target detection model at a learning rate lower than that of the previous update process.

[0187] In an exemplary embodiment, the system further includes a model validation module, which can be used to: obtain the validation score of the object detection model on the validation dataset at the end of each training round; update the model checkpoint with the object detection model of the current training round when the validation score is better than the historical best validation score; and stop parameter updating when the validation score has not exceeded the historical best validation score for a consecutive preset number of rounds.

[0188] In one exemplary embodiment, the preset category is a fabric defect category, and the target detection model is a fabric defect detection model.

[0189] Each module in the aforementioned model optimization system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in an electronic device, or stored in the memory of the electronic device as software, so that the processor can call and execute the corresponding operations of each module.

[0190] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the model optimization methods described above.

[0191] In one exemplary embodiment, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the model optimization methods described in the above embodiments.

[0192] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the model optimization methods described in the above embodiments.

[0193] Indicatively, such as Figure 10 As shown, Figure 10 This is a schematic diagram of the internal structure of an electronic device 1000 provided in an embodiment of this application. The electronic device 1000 can be provided as a server. (Refer to...) Figure 10 The electronic device 1000 includes a processor 1002, which further includes one or more processors, and memory resources represented by memory 1001 for storing instructions executable by the processor 1002, such as a computer program. The computer program stored in memory 1001 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 1002 is configured to execute instructions to perform the model optimization methods of any of the above embodiments. The electronic device 1000 can operate on an operating system stored in memory 1001, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0194] The electronic device 1000 may further include a power supply component 1003 configured to perform power management of the electronic device 1000, a wired or wireless network interface 1004 configured to connect the electronic device 1000 to a network, and an input / output (I / O) interface 1005. Wireless operation may be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a model optimization method. The display unit 1007 of the electronic device is used to form a visually visible image and may be a display screen, a projection system, or a virtual reality imaging system. The display screen may be an LCD screen or an e-ink display screen. The input system 1006 of the electronic device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the casing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0195] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0199] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model optimization method, characterized in that, include: Obtain performance feedback information of the object detection model on the validation dataset; Based on the performance feedback information, the learning difficulty of the target detection model in each preset category is determined, and based on the learning difficulty, at least one difficult category is determined from each preset category; Based on the learning difficulty of each difficulty category, determine the amount of synthetic data corresponding to each difficulty category; Based on the amount of synthetic data, synthetic samples are generated for each of the aforementioned difficulty categories, and the synthetic samples carry annotation information; The synthetic samples are fused with real training samples, and the parameters of the target detection model are updated using the fused samples.

2. The method according to claim 1, characterized in that, The performance feedback information includes the prediction results of the target detection model on the validation dataset; The step of determining the learning difficulty of the target detection model in each preset category based on the performance feedback information includes: Based on the prediction results, the false detection information for each preset category is statistically analyzed, wherein the false detection information includes the missed detection rate and / or the inter-class confusion rate; The learning difficulty of each preset category is determined based on the false negative rate and / or the inter-class confusion rate.

3. The method according to claim 2, characterized in that, The step of determining the learning difficulty of each preset category based on the false negative rate and / or the inter-class confusion rate includes: The total error rate is determined based on the false negative rate, and the inter-class confusion penalty is determined based on the inter-class confusion rate. The learning difficulty of each preset category is determined by the weighted sum of the total error rate of each preset category and the inter-class confusion penalty.

4. The method according to claim 1, characterized in that, Determining the amount of synthetic data corresponding to each of the aforementioned difficulty categories includes: Obtain the number of real samples for each of the aforementioned difficulty categories in the real training set; Based on the number of real samples, determine the frequency compensation amount for each of the difficulty categories; Based on the learning difficulty level, determine the difficulty compensation amount for each of the aforementioned difficulty categories; The amount of synthesized data for each difficulty category is determined by the weighted sum of the frequency compensation amount and the difficulty compensation amount.

5. The method according to claim 4, characterized in that, The determination of the amount of synthetic data for each of the aforementioned difficulty categories includes: The frequency compensation amount for each difficulty category is determined based on the difference between the preset reference number and the actual sample number; wherein, when the actual sample number is greater than or equal to the preset reference number, the frequency compensation amount is zero. The difficulty compensation amount for each difficulty category is determined based on the product of the learning difficulty and the preset reference quantity. The amount of synthetic data for each difficulty category is determined by the weighted sum of the frequency compensation amount and the difficulty compensation amount for each difficulty category.

6. The method according to claim 1, characterized in that, The step of generating synthetic samples for each of the aforementioned difficulty categories based on the amount of synthetic data includes: Obtain the background image and its corresponding spatial constraint information; The spatial constraint information is used as a condition input to control the preset generation model to synthesize sample content that conforms to the corresponding difficulty category features within a specified area of ​​the background image; Based on the spatial constraint information, the annotation information of the sample content is generated.

7. The method according to claim 1, characterized in that, The step of fusing the synthetic samples with real training samples and using the fused samples to update the parameters of the target detection model includes: In each training mini-batch, samples are taken from the real training sample set and the synthetic sample set according to a preset participation ratio; The sampled real training samples and synthetic samples are concatenated to form the current training mini-batch; Based on the current training mini-batch, determine the joint loss to update the parameters of the object detection model.

8. The method according to claim 7, characterized in that, The determination of joint loss includes: Based on the subset of real training samples in the current training mini-batch, determine the detection loss for the subset of real training samples; Determine the detection loss for the synthetic sample subset based on the synthetic sample subset in the current training mini-batch; The joint loss is determined by a weighted sum of the detection loss of the real training sample subset and the detection loss of the synthetic sample subset; wherein the weight of each loss is determined based on the proportion of the corresponding samples in the current training mini-batch.

9. A model optimization system, characterized in that, include: The acquisition module is used to obtain performance feedback information of the object detection model on the validation dataset. The diagnostic module is used to determine the learning difficulty of the target detection model in each preset category based on the performance feedback information, and to determine at least one difficult category from each preset category based on the learning difficulty. The quota allocation module is used to determine the amount of synthetic data corresponding to each difficulty category based on the learning difficulty of each difficulty category; A synthesis module is used to generate synthetic samples for each of the difficulty categories based on the amount of synthetic data, wherein the synthetic samples carry annotation information; An optimization module is used to fuse the synthetic samples with real training samples and use the fused samples to update the parameters of the target detection model.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-8.

Images