A method and device for content management and control based on mixed enhancement of large models

Abstract

The embodiment of the application provides a kind of based on mixed enhancement's big model content control method and device, comprising: receiving original sample picture containing multiple abnormal categories, original sample picture is mixed enhanced and generates mixed sample picture, mixed sample picture and original sample picture are input model as training sample and are trained, based on first probability distribution and second probability distribution, output prediction classification result and predicted sample identification, and according to prediction classification result and predicted sample identification, generate final abnormal category probability distribution.The present application introduces mixed enhancement technology, double-head output structure and confidence threshold mechanism, solves the problem that existing image classification model is insufficient in the open set scene to unknown class identification ability, significantly improves the discrimination accuracy and robustness of model to abnormal category, reduces false positive rate.

Description

Technical Field

[0001] This application relates to the technical field of network security and data security, and in particular to a method and apparatus for managing large model content based on hybrid enhancement. Background Technology

[0002] With the rapid development of internet multimedia technology, the security review of image content has become a key aspect of cyberspace governance. Existing image detection and recognition technologies are mainly based on deep learning models, which extract and classify features from input images to identify whether they contain sensitive or inappropriate content.

[0003] However, existing image classification models are typically trained and deployed based on the "closed set" assumption. That is, during model training, the training dataset contains only predefined known categories (such as normal, abnormal, etc.), and the model is optimized to force input samples to be mapped to one of these known categories. In real-world, open-ended applications, the distribution of test data often differs from that of training data, and input images inevitably contain unknown category samples not seen during training.

[0004] Because existing models lack the ability to identify and reject unknown categories, when encountering images of unknown categories, they still forcefully classify them into a known category based on feature similarity, leading to a high false positive rate. For example, in scenarios involving the detection of prohibited items, if the training set only contains images of anomalous categories and not benign images such as "farm implements" or "tools," existing models are very likely to incorrectly classify "farm implements" or "tools" images as anomalous categories during actual detection due to the similarity of visual features (such as both containing long handles or metal parts).

[0005] The lack of this "open set recognition" capability not only reduces the model's generalization ability and robustness in complex real-world environments, but also leads to a large number of normal images being misclassified as sensitive images. This significantly increases the cost of manual review, impacting user experience and the efficiency of the review system. Therefore, how to improve the model's adaptability to unknown samples without reducing the detection accuracy of known sensitive categories, and avoid forcibly classifying unknown benign samples into known sensitive categories, has become a pressing technical problem in the field of image content security detection. Summary of the Invention

[0006] The purpose of this application is to provide a method and apparatus for large model content management based on hybrid enhancement, so as to solve the problem of inaccurate classification in existing image detection methods and achieve the technical effect of improving detection accuracy. The specific technical solution is as follows: In a first aspect, embodiments of the present invention provide a large model content management method based on hybrid enhancement. The method includes: receiving original sample images containing multiple anomaly categories, the original sample images having a first probability distribution representing their respective anomaly categories; performing hybrid enhancement on the original sample images to obtain hybrid sample images, wherein the hybrid sample images have a second probability distribution representing their respective anomaly categories, the second probability distribution being associated with the first probability distribution; inputting the hybrid sample images and the original sample images as training samples into a model for training, outputting a predicted classification result and a predicted sample identifier based on the first and second probability distributions, the predicted classification result representing the probability distribution of the training samples belonging to anomaly categories, and the predicted sample identifier representing the probability that the training samples belong to the original sample images; obtaining a final anomaly category probability distribution based on the predicted classification result and the predicted sample identifier, wherein the final anomaly category probability distribution represents the probability that the test image input into the model belongs to any anomaly category.

[0007] Optionally, the process of blending and enhancing the original sample image to obtain a blended sample image includes: extracting a first sample image and a second sample image from the original sample image; setting the first sample image as the base image; extracting target pixel blocks from the second sample image as an overlay layer; setting the pixels of the first sample image to a first transparency parameter X at the pixel positions corresponding to the target pixel blocks, and setting the target pixel blocks of the second sample image to a second transparency parameter (1-X); and overlaying the first sample image and the second sample image according to the first transparency parameter X and the second transparency parameter (1-X) to generate a blended sample image.

[0008] Optionally, outputting the predicted classification result and predicted sample identifier based on the first probability distribution and the second probability distribution includes: calculating the degree of overlap between the target pixel block and the preset feature region in the first sample image. The preset feature region is the area in the first sample image where the pixel value is not the background color; based on the degree of overlap The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Generate the probability distribution vector of the mixed sample images As the predicted classification result; set the predicted sample label for the mixed sample image, and mark the original sample probability as a preset value to indicate that the current sample is a mixed sample image generated by the mixed enhancement.

[0009] Optionally, the method includes: performing blending enhancement on the original sample image to obtain a blended sample image, including: extracting a first sample image and a second sample image from the original sample image, setting the first sample image as the base image, and cropping a target pixel block from the second sample image; directly overlaying and pasting the target pixel block onto the corresponding position of the first sample image to obtain the blended sample image.

[0010] Optionally, outputting the predicted classification result and the predicted sample identifier based on the first probability distribution and the second probability distribution includes: identifying a preset feature region in the first sample image, and calculating the coincidence degree between the target pixel block and the preset feature region ; based on the coincidence degree and the class confidence of the abnormal class to which the target pixel block belongs in the second sample image generating the probability distribution vector of the mixed sample image , as the predicted classification result; setting a predicted sample identifier for the mixed sample image, and marking the original sample probability as a preset value, to indicate that the current sample is a mixed sample image generated by mixing enhancement.

[0011] Optionally, the final abnormal class probability distribution is:

[0012] where T is the temperature scaling parameter, represents the predicted classification result, represents the predicted sample identifier, using as the confidence gating factor, when the value is small, it indicates that the training sample is a mixed sample image; when the value is large, the predicted classification result of the training sample is retained.

[0013] Optionally, after obtaining the final abnormal class probability distribution according to the predicted classification result and the predicted sample identifier, a classification loss function is constructed, and the classification loss function calculates the final abnormal class probability distribution based on the Kullback-Leibler divergence and the probability distribution vector of the mixed sample image the distribution difference therebetween; , wherein the probability distribution vector of the mixed sample image is a probability distribution dynamically generated during image mixing enhancement, the value of which is determined based on the pixel coincidence degree and the class confidence of the mixed sample image, and is used to characterize the weight proportion of each abnormal class feature in the mixed sample; when the Loss value starts to oscillate or is less than the training stop threshold , the training is stopped.

[0014] Optionally, the maximum value of the final abnormal class probability distribution of the image to be tested is acquired ; when the maximum value is less than the open set determination threshold T1, it is determined that the image to be tested does not belong to any abnormal class; when the maximum value is greater than the class determination threshold T2 (T1<T2), it is determined that the image to be tested belongs to a definite abnormal class; when the maximum value If the value is greater than the open set judgment threshold T1 and less than the category judgment threshold T2, the category of the current image to be tested cannot be determined. A binary classification model is used for secondary recognition, which is used to identify and output the category of the image to be tested.

[0015] Secondly, embodiments of the present invention provide a large model content management device based on hybrid enhancement. The device includes: a receiving module for receiving original sample images containing multiple anomaly categories, the original sample images having a first probability distribution representing their respective anomaly categories; a hybrid enhancement module for performing hybrid enhancement on the original sample images to obtain hybrid sample images, wherein the hybrid sample images have a second probability distribution representing their respective anomaly categories, the second probability distribution being associated with the first probability distribution; a training module for inputting the hybrid sample images and the original sample images as training samples into a model for training, and outputting a predicted classification result and a predicted sample identifier based on the first probability distribution and the second probability distribution, the predicted classification result representing the probability distribution of the training sample belonging to an anomaly category, and the predicted sample identifier representing the probability that the training sample belongs to the original sample image; and a probability distribution calculation module for obtaining a final anomaly category probability distribution based on the predicted classification result and the predicted sample identifier, wherein the final anomaly category probability distribution representing the probability that the test image input into the model belongs to any anomaly category.

[0016] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, describes any of the methods described in the first aspect.

[0017] Beneficial effects of the embodiments in this application: This application provides a method, apparatus, and electronic device for managing large model content based on hybrid enhancement. It generates diverse training samples (including mixed samples and original samples) through hybrid enhancement and combines this with a dual-head output structure (predicted classification result and predicted sample identifier) ​​to generate the final anomaly category probability distribution. This allows the model to dynamically adjust the confidence output for mixed and original samples, thereby achieving more accurate probability distribution calculation in open set scenarios. By introducing hybrid enhancement technology, a dual-head output structure, and a temperature scaling parameter gating mechanism, it solves the technical problems of insufficient ability to identify unknown categories and high false positive rates in existing image classification models in open set scenarios.

[0018] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0020] Figure 1 A flowchart of an image data processing method provided in an embodiment of the present invention; Figure 2 A feature contribution heatmap for generating hybrid sample images using hybrid enhancement in this embodiment of the invention; Figure 3 This is a schematic diagram of the image data processing device according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0022] The hybrid enhancement-based large-model content control method provided in this application can be applied to the field of image detection to detect and control abnormal images. The multi-classification model in this application can detect the probability of each image belonging to each of multiple categories, thereby determining the abnormal category of each image. If no category can be determined, a binary classification model is used for secondary filtering to determine the abnormal category of each image. For images of abnormal categories, they are synchronized to edge control AI devices for management, and actions are taken based on the detection results and handling strategies, such as transmission blocking, image replacement, screen shutdown, and device shutdown.

[0023] To address the inaccuracy of image classification in existing image detection methods, this application provides a large-model content management method, apparatus, and electronic device based on hybrid enhancement. The large-model content management method based on hybrid enhancement in this application identifies various categories of anomalous images by training a multi-input multi-output image recognition model. For complex images containing multiple anomalous features, this application effectively overcomes the shortcomings of traditional models, such as missed detections or misclassifications, significantly improving the accuracy of image content recognition and the precision of anomalous classification, thus achieving the technical objective of optimizing image processing effects.

[0024] Figure 1 This is a flowchart of a large model content management method based on hybrid enhancement, according to an embodiment of this application. Figure 1As shown in the embodiments of this application, the large model content management method based on hybrid enhancement may include the following steps: Step S101: Receive raw sample images containing multiple anomaly categories, wherein the raw sample images have a first probability distribution representing the anomaly category to which they belong; This application constructs a multi-input, multi-output (MIMO) multi-classification model. The model input consists of images from various anomaly categories (normal images are not included in subsequent training). Each training batch simultaneously contains images from all K anomaly categories, with M images for each category, instead of training only one category. This ensures that the model sees the decision boundaries of multiple categories simultaneously in the same gradient update step, avoiding the model focusing only on features of a single category. The goal is to enable the model to simultaneously identify multiple anomaly categories and improve its generalization ability through a "hybrid augmentation" technique.

[0025] Each original sample image has a first probability distribution representing its anomaly category, which can be obtained based on multiple binary classification models. The method for constructing multiple binary classification models is as follows: 1. Dataset Construction and Partitioning The original image data is acquired and categorized. The categorization labels include a normal category and K abnormal categories, for a total of K+1 categories. The labeled dataset is then divided into a training set, a validation set, and a test set according to a preset ratio. In a preferred embodiment of this application, the preset ratio is 8:1:1, meaning the training set accounts for 80%, the validation set for 10%, and the test set for 10%.

[0026] 2. Construction of Binary Classification Recognition Model Construct a binary classification recognition model based on a convolutional neural network (CNN). The backbone of the binary classification recognition model can be a ResNet series network or other deep learning networks with feature extraction capabilities.

[0027] The binary classification recognition model is configured to receive an input image and output a predicted label indicating whether the image belongs to the "normal category" or the "specific anomaly category". It should be noted that this solution employs a "one-vs-rest" modeling strategy, meaning that for each anomaly category, a separate binary classification model is built to distinguish between "normal images" and "images belonging to that specific anomaly category".

[0028] 3. Model Training Based on the training set obtained from the previous division, a corresponding binary classification recognition model is trained for each anomaly category.

[0029] Specifically, if there are K anomaly categories, then K binary classification models are trained. The training objective of the i-th model (1≤i≤K) is to label images belonging to the i-th anomaly category as positive samples and images belonging to the normal category as negative samples. The model parameters are updated through backpropagation until the model's performance on the validation set converges, resulting in K trained binary classification recognition models.

[0030] 4. Pixel contribution identification (feature visualization) The feature maps of the intermediate layer of the binary classification recognition model are obtained. These feature maps are then weighted and summed using preset weights, and processed using a Rectified Linear Unit (ReLU) activation function to filter out negative signals, retaining only feature regions that positively contribute to the target category. Subsequently, the processed feature maps are upsampled to ensure their spatial resolution matches the size of the original input image, facilitating pixel-level correspondence analysis.

[0031] In a preferred embodiment of this application, the heatmap is generated using a Gradient Weighted Class Activation Map (Grad-CAM) algorithm. Specifically, the gradient information of the predicted score of the target category relative to the output features of the last convolutional layer of the model is calculated. This gradient information is then used as weights to weight the feature map, thereby generating a feature contribution heatmap, as shown in the image. Figure 2 As shown.

[0032] The pixel values ​​in the feature contribution heatmap quantify the contribution of a pixel region at a corresponding spatial location in the input image to the model's determination of the current specific category. Higher pixel values ​​indicate a greater influence of that region on the classification result; conversely, lower values ​​indicate a smaller or no influence.

[0033] 5. Key Feature Block Extraction Based on the generated heatmap, key feature blocks of different levels are extracted according to pixel contribution values. Multiple contribution threshold levels are set to divide the heatmap into several regions. In one embodiment of this application, five contribution levels [0, 1, 2, 3, 4] are set: Level 0: Corresponds to the area with the lowest contribution in the heatmap (e.g., background area or inactive area); Levels 1 to 4: These correspond to regions in the heatmap where the contribution increases sequentially (e.g., regions with heat values ​​greater than the green threshold, yellow threshold, red threshold, and dark red threshold, respectively).

[0034] Through the above classification, the original image is divided into pixel blocks corresponding to different contribution levels. Each pixel block represents the contribution strength of that region to the correct identification of the current image into the corresponding category.

[0035] 6. Calculation of key feature block probability values Pixel blocks of different levels are processed, retaining pixels in the corresponding level while masking other areas. The processed image is then input into a pre-trained binary classification model for forward inference calculation, yielding the model's output confidence score (Conf). This confidence score (Conf) is used as the probability value for the corresponding pixel block. This probability value quantifies the degree of certainty regarding the classification result for an image region at a specific contribution level, and is used for subsequent image anomaly detection.

[0036] Through the above steps, multiple binary classification models can be obtained. Each binary classification model can identify an abnormal category of image and obtain the probability of each original sample image, thus obtaining the first probability distribution.

[0037] Through the above steps, this application can not only accurately identify multiple anomaly categories, but also locate key image regions that lead to model judgments through pixel-level contribution analysis, thereby improving the interpretability of the model and its adaptability to complex scenes.

[0038] Step S103: Perform blending enhancement on the original sample image to obtain a blended sample image, wherein the blended sample image has a second probability distribution representing its anomaly category, and the second probability distribution is associated with the first probability distribution; To further improve the robustness and generalization performance of the model in complex scenarios, this application introduces a hybrid enhancement strategy and a batch-level relational inference mechanism during the model training phase. The specific implementation process is as follows: Based on the obtained feature contribution heatmap and key feature blocks, regions with contribution values ​​higher than a preset threshold are selected as "high contribution mixed blocks".

[0039] When constructing training batches, for the current input image of the target category, high-contribution feature blocks are extracted from images of other categories and blended with the feature regions of the current image. This generates hybrid sample images containing multi-category semantic information. This process differs from traditional simple image stitching or parallel processing, aiming to simulate the complex situation of abnormal features intertwined with normal backgrounds in real-world scenes.

[0040] To address the poor enhancement results caused by the randomness of sample selection during hybrid augmentation, a batch-level graph neural network (GNN) module is introduced. Each sample in the current batch is treated as a node in a graph, and the graph neural network aggregates the relational features between samples. Through the message passing mechanism of the GNN, the potential associations between samples are inferred, thereby guiding the hybrid weights and position selection of feature blocks. This allows the model to not only focus on the features of a single image but also to utilize the global relationships between samples within a batch for inference, enhancing the discriminative power of the feature representation.

[0041] 1) Construct a sample relationship diagram Samples are represented as points, and the relationships between samples are represented as edges. Edges include confidence edges and heatmap relative position edges: within each category, samples from different categories are connected according to the confidence interval [0.6 0.7 0.8 0.9 1.0] as confidence edges; samples are divided into N*N pixel blocks, and the heatmap calculates the relative position of the pixel block on the sample according to the contribution interval (e.g., all are central areas containing red pixels), and samples with the same relative position in different categories are connected as heatmap edges.

[0042] 2) Sample selection For the samples to be mixed, take the top-K samples with confidence edges, and use the mixup method to perform mixing enhancement (transparency overlay) to generate K enhanced samples.

[0043] For the samples to be mixed, take the Top-K samples with heatmap edges, and use the cutmix method to perform mixing enhancement (pixel block pasting) to generate K*5 enhanced samples (each sample has 5 pixel contribution blocks).

[0044] Instead of using a traditional single hard label for the generated mixed sample images, new vector labels are dynamically generated based on the probability values ​​of image pixel blocks. .

[0045] Specifically, The dimension of the vector is consistent with the number of anomaly categories, and its value is calculated by weighting the proportion of other category feature blocks mixed into the current image and their corresponding confidence probability values. This vector label can more accurately reflect the distribution of each category feature in the mixed image and serve as a supervisory signal to guide model learning.

[0046] The advantage of this approach is that it allows the model to see more "mixed sample images" during training. For example, it stitches together the regions of Class A outlier images (red heatmap) and the regions of Class B outlier images (yellow heatmap). This allows the model to learn to judge the features of "mixed sample images," thereby improving the model's robustness and accuracy.

[0047] Step S105: Input the mixed sample image and the original sample image as training samples into the model for training. Output the prediction classification result and the prediction sample label based on the first probability distribution and the second probability distribution. The prediction classification result is used to represent the probability distribution of the training sample belonging to the abnormal category, and the prediction sample label is used to identify the probability of the training sample belonging to the original sample image. A multi-classification model is trained by inputting mixed sample images and original sample images as training samples. The model outputs a predicted classification result and a predicted sample identifier based on a first probability distribution (i.e., the probability vector of the abnormal category to which the sample belongs) and a second probability distribution (i.e., the probability scalar of the sample being the original sample). The predicted classification result is the probability distribution vector of the training sample belonging to K abnormal categories, and the predicted sample identifier is the probability value of the training sample belonging to the original sample (1 represents the original sample, and 0 represents the mixed augmented sample).

[0048] Step S107: Obtain the final anomaly category probability distribution based on the predicted classification results and predicted sample identifiers. The final anomaly category probability distribution is used to represent the probability that the test image input to the model belongs to any anomaly category.

[0049] The final anomaly category probability distribution is obtained through a dual-head output structure based on the predicted classification results and predicted sample identifiers. The final anomaly category probability distribution is generated as follows: the predicted classification results and predicted sample identifiers are multiplied by a temperature scaling parameter T, where the temperature scaling parameter T serves as a confidence gating mechanism. When the predicted sample identifier value is small (indicating that the sample image is a mixed sample image), the probability values ​​of all anomaly categories are suppressed, thereby realizing the calculation of the anomaly category probability distribution in the open set scenario. This distribution is used to characterize the probability that the test image of the input model belongs to any anomaly category.

[0050] The final probability distribution of the anomaly categories is as follows:

[0051] Where T is the temperature scaling parameter, Indicates the predicted classification result. Indicates the predicted sample identifier, As a confidence gating factor, when When the value is small, it indicates that the training samples are mixed sample images; when When the value is large, the predicted classification results of the training samples are retained.

[0052] This application's embodiments increase the diversity of anomaly categories learned by the model by using both original and mixed sample images during the training of a multi-input multi-output image detection model, thus enhancing the model's robustness. Simultaneously, the dual-head output forces the ResNet backbone network to learn more robust features by judging whether an image is "original" or "mixed" during training. These features participate in backpropagation, helping the model training to focus more on the classification of real images. Because of the probability distribution of mixed samples and training samples belonging to anomaly categories, using the trained model for image detection can overcome the problem of inaccurate identification of unknown categories in existing technologies, improving the accuracy of image anomaly category recognition.

[0053] Optionally, after obtaining the final anomaly category probability distribution based on the predicted classification results and predicted sample identifiers, the method further includes: Construct a classification loss function, which calculates the final anomaly class probability distribution based on the Kullback-Leibler divergence. The probability distribution vector of the mixed sample images The differences in distribution between them;

[0054] Among them, the probability distribution vector of the mixed sample images The probability distribution is dynamically generated during image blending enhancement. Its value is determined based on the pixel overlap and class confidence of the blended sample images and is used to characterize the weight ratio of each abnormal class feature in the blended sample. When the loss value starts to fluctuate or falls below the training stopping threshold If the training stops, then training should be stopped.

[0055] By introducing dynamic generation As a supervisory signal, combined with the KL divergence loss function, it can effectively constrain the model to accurately model the probability distribution of anomaly categories in mixed samples. Meanwhile, the gating mechanism of the temperature scaling parameter T and... The joint optimization further enhances the robustness and generalization ability of the model in open set anomaly detection tasks.

[0056] The hybrid sample images in this application employ multiple hybrid enhancement schemes, increasing the complexity of the samples and further improving the effectiveness of the model. This application embodiment uses two methods to perform hybrid enhancement on the original sample images.

[0057] (1) Generation based on probability mixup The process of blending and enhancing the original sample images to obtain a blended sample image includes: extracting a first sample image and a second sample image from the original sample images; setting the first sample image as the base image; extracting target pixel blocks from the second sample image as an overlay layer; setting the pixels of the first sample image to a first transparency parameter X at the pixel positions corresponding to the target pixel blocks, and setting the target pixel blocks of the second sample image to a second transparency parameter (1-X); and overlaying the first sample image and the second sample image according to the first transparency parameter X and the second transparency parameter (1-X) to generate a blended sample image.

[0058] Outputting the predicted classification result and predicted sample label based on the first probability distribution and the second probability distribution includes: calculating the degree of overlap between the target pixel block and the preset feature region in the first sample image. The preset feature region is the region in the first sample image where the pixel value is not the background color; based on the degree of overlap... The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Generate the probability distribution vector of the mixed sample images As the predicted classification result; set the predicted sample label for the mixed sample image, and mark the original sample probability as a preset value to indicate that the current sample is a mixed sample image generated by the mixed enhancement.

[0059] A first sample image and a second sample image are selected from the original sample image set as the mixing base. The first sample image is set as the base image, and the second sample image is used to extract the regions whose pixel values ​​are not the background color through an object detection algorithm to form target pixel blocks as the overlay layer.

[0060] At the pixel location corresponding to the target pixel block, a first transparency parameter X is set for the pixels in the first sample image, and a second transparency parameter (1-X) is set for the target pixel block in the second sample image, where X∈(0,1). The transparency parameter can be set based on a preset linear decay function or a random sampling strategy to control the weight ratio of the original sample features and the superimposed sample features in the mixed sample. The first sample image and the second sample image are pixel-level superimposed according to the first transparency parameter X and the second transparency parameter (1-X) to generate a mixed sample image. This mixed sample image simultaneously retains the background features of the base image and the target features of the superimposed layer, forming a semantically related composite image sample.

[0061] Based on pixel overlap The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Dynamically generate probability distribution vectors of mixed sample images . Specifically, The generation process includes: according to Calculate the feature weight ratio between the base image and the overlay layer. = × X and = (1- ) ×(1-X); The probability distribution of anomaly categories in the base map Anomaly category probability distribution of overlay layers According to weight and By performing weighted fusion, we obtain = × + × ; The weighted probability distribution is normalized to ensure... The numerical range of is [0,1]. A predicted sample identifier is set for the mixed sample image. The original sample probability is marked with a preset value S=0 to indicate that the current sample is a mixed sample image generated by bluffing enhancement; for the original sample image, the predicted sample identifier is set to S=1. This identifier is used to distinguish sample types during subsequent training to optimize the model output in conjunction with the gating mechanism of the temperature scaling parameter T.

[0062] In this embodiment of the application, pixel overlap is introduced. The joint calculation with the transparency parameter X, combined with the category confidence score. Dynamically generated This approach can accurately characterize the feature weight ratios of each anomaly category in mixed samples. Furthermore, the synergistic effect of setting the predicted sample identifier S and the dual-head output structure significantly improves the model's ability to discriminate mixed augmented samples, solving the technical challenge of unbalanced sample distribution in open set anomaly detection.

[0063] (2) Generation based on probability cutmix The process of blending and enhancing the original sample images to obtain blended sample images involves: extracting a first sample image and a second sample image from the original sample images; setting the first sample image as the base image; cropping target pixel blocks from the second sample image; and directly overwriting and pasting the target pixel blocks onto the corresponding positions of the first sample image to obtain the blended sample image.

[0064] Outputting the predicted classification result and predicted sample label based on the first probability distribution and the second probability distribution includes: identifying the preset feature region in the first sample image and calculating the degree of overlap between the target pixel block and the preset feature region. Based on the degree of overlap And the category confidence of the target pixel block in the anomaly category of the second sample image. Probability distribution vector for generating mixed sample images This serves as the predicted classification result. A predicted sample identifier is set for the mixed sample image, and the original sample probability is marked with a preset value to indicate that the current sample is a mixed sample image generated by the mixture enhancement.

[0065] The first and second sample images are selected from the original sample image set as the mixing base. The first sample image is set as the base image, and the second sample image is used to extract regions whose pixel values ​​are not background colors using an object detection algorithm, forming target pixel blocks as the overlay layer. These target pixel blocks are directly overlaid onto the corresponding positions in the first sample image to generate the mixed sample image. This overlay operation is achieved through coordinate mapping to ensure that the target pixel blocks are aligned with the semantic regions of the base image, thereby generating a semantically coherent composite image sample.

[0066] Identify a preset feature region in the first sample image. The preset feature region is a region in the first sample image whose pixel value is not the background color; its boundary can be determined using a semantic segmentation algorithm or an edge detection algorithm. Calculate the pixel overlap between the target pixel block and the preset feature region. .

[0067] Based on overlap And the category confidence of the target pixel block in the anomaly category of the second sample image. Dynamically generate probability distribution vectors of mixed sample images . Specifically, The generation process includes: according to Calculate the coverage weight of the target pixel block on the base map features. = × ; The probability distribution of anomaly categories in the base map Anomaly category probability distribution of target pixel block According to weight By performing weighted fusion, we obtain = (1 - ) × + × ; The weighted probability distribution is normalized to ensure... has a numerical range of [0,1]. A prediction sample identifier is set for the mixed sample image. The original sample probability is marked as a preset value S=0, to indicate that the current sample is a mixed sample image generated by mixed augmentation; for the original sample image, the prediction sample identifier is set to S=1. This identifier is used to distinguish sample types in the subsequent training process, so as to cooperate with the gating mechanism of the temperature scaling parameter T to optimize the model output.

[0068] In the embodiments of the present application, by introducing the pixel coincidence degree and the class confidence for joint calculation, combined with the semantic coverage strategy to dynamically generate , the feature weight proportion of each anomaly class in the mixed sample can be accurately represented. Meanwhile, the synergistic effect of the setting of the prediction sample identifier S and the dual-head output structure significantly improves the model's discrimination ability for mixed augmented samples, and solves the technical problem of unbalanced sample distribution in open-set anomaly detection.

[0069] Optionally, in the actual use process of the multi-input multi-output model according to the embodiments of the present application, it is first judged which anomaly class the test image belongs to, and if the judgment cannot be made, a binary classification model is used for recognition. Optionally, the method further comprises: acquiring the maximum value of the final anomaly class probability distribution of the test image ; when the maximum value is smaller than the open-set judgment threshold T1, it is considered that the test image does not belong to any anomaly class; when the maximum value is greater than the class judgment threshold T2 (T1<T2), it is considered that the test image belongs to a determined anomaly class; when the maximum value is greater than the open-set judgment threshold T1 and smaller than the class judgment threshold T2, the class to which the current test image belongs cannot be determined, and the binary classification model is adopted for secondary recognition, wherein the binary classification model is used for recognizing and outputting the class of the test image.

[0070] In the case of fuzzy judgment, a preset binary classification model is called to perform secondary recognition on the test image. The binary classification model is an independently trained auxiliary model, whose inputs are the feature vector of the test image and the original anomaly class probability distribution , and whose output is the binary classification result of "anomaly class" or "normal class". In the present invention, by introducing the synergistic mechanism of the multi-threshold judgment strategy and the binary classification model, the fine-grained classification of the open-set anomaly detection task is realized. Wherein: Threshold judgment logic: by setting the gradient thresholds of T1 and T2, the anomaly class probability distribution is divided into three decision intervals of "normal class", "determined anomaly class" and "fuzzy region", which improves the processing ability of the model for low-confidence samples; Synergistic effect of binary classification models: For samples in ambiguous regions, binary classification models introduce additional discrimination criteria (such as joint information of feature vectors and probability distributions) to compensate for the classification uncertainty of the main model in open set scenarios; Linkage with temperature scaling parameter T: The gating mechanism of temperature scaling parameter T forms a closed loop with the current judgment logic, ensuring the final probability distribution of anomaly categories. The confidence distribution conforms to the statistical characteristics of threshold determination.

[0071] The large model content management method based on hybrid enhancement provided in this application training a multi-input multi-output model, which improves the generalization ability of the model by diversifying the training samples. When using this model for image recognition, the abnormal category to which the image belongs can be given according to the output abnormal category probability. When the output result of the model is a decision on a fuzzy region, a binary classification model is introduced to further confirm the abnormal category to which the image belongs, thereby improving the recognition accuracy.

[0072] This application also provides a large model content management device based on hybrid enhancement.

[0073] Figure 3 This is a schematic diagram of a large model content management device based on hybrid enhancement, according to an embodiment of this application. Figure 3 As shown, the large model content management device based on hybrid enhancement includes: The receiving module 30 is used to receive original sample images containing multiple anomaly categories, wherein the original sample images have a first probability distribution representing the anomaly category to which they belong; This application constructs a multi-input, multi-output (MIMO) multi-classification model. The model input consists of images from various anomaly categories (normal images are not included in subsequent training). Each training batch simultaneously contains images from all K anomaly categories, with M images for each category, instead of training only one category. This ensures that the model sees the decision boundaries of multiple categories simultaneously in the same gradient update step, avoiding the model focusing only on features of a single category. The goal is to enable the model to simultaneously identify multiple anomaly categories and improve its generalization ability through a "hybrid augmentation" technique.

[0074] Each original sample image has a first probability distribution representing its anomaly category, which can be obtained based on multiple binary classification models. The method for constructing multiple binary classification models is as follows: 1. Dataset Construction and Partitioning The original image data is acquired and categorized. The categorization labels include a normal category and K abnormal categories, for a total of K+1 categories. The labeled dataset is then divided into a training set, a validation set, and a test set according to a preset ratio. In a preferred embodiment of this application, the preset ratio is 8:1:1, meaning the training set accounts for 80%, the validation set for 10%, and the test set for 10%.

[0075] 2. Construction of Binary Classification Recognition Model Construct a binary classification recognition model based on a convolutional neural network (CNN). The backbone of the binary classification recognition model can be a ResNet series network or other deep learning networks with feature extraction capabilities.

[0076] The binary classification recognition model is configured to receive an input image and output a predicted label indicating whether the image belongs to the "normal category" or the "specific anomaly category". It should be noted that this solution employs a "one-vs-rest" modeling strategy, meaning that for each anomaly category, a separate binary classification model is built to distinguish between "normal images" and "images belonging to that specific anomaly category".

[0077] 3. Model Training Based on the training set obtained from the previous division, a corresponding binary classification recognition model is trained for each anomaly category.

[0078] Specifically, if there are K anomaly categories, then K binary classification models are trained. The training objective of the i-th model (1≤i≤K) is to label images belonging to the i-th anomaly category as positive samples and images belonging to the normal category as negative samples. The model parameters are updated through backpropagation until the model's performance on the validation set converges, resulting in K trained binary classification recognition models.

[0079] 4. Pixel contribution identification (feature visualization) The feature maps of the intermediate layer of the binary classification recognition model are obtained. These feature maps are then weighted and summed using preset weights, and processed using a Rectified Linear Unit (ReLU) activation function to filter out negative signals, retaining only feature regions that positively contribute to the target category. Subsequently, the processed feature maps are upsampled to ensure their spatial resolution matches the size of the original input image, facilitating pixel-level correspondence analysis.

[0080] In a preferred embodiment of this application, the heatmap is generated using the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm. Specifically, the gradient information of the predicted score of the target category relative to the output features of the last convolutional layer of the model is calculated, and this gradient information is used as weights to weight the feature map, thereby generating a feature contribution heatmap.

[0081] The pixel values ​​in the feature contribution heatmap quantify the contribution of a pixel region at a corresponding spatial location in the input image to the model's determination of the current specific category. Higher pixel values ​​indicate a greater influence of that region on the classification result; conversely, lower values ​​indicate a smaller or no influence.

[0082] 5. Key Feature Block Extraction Based on the generated heatmap, key feature blocks of different levels are extracted according to pixel contribution values. Multiple contribution threshold levels are set to divide the heatmap into several regions. In one embodiment of this application, five contribution levels [0, 1, 2, 3, 4] are set: Level 0: Corresponds to the area with the lowest contribution in the heatmap (e.g., background area or inactive area); Levels 1 to 4: These correspond to regions in the heatmap where the contribution increases sequentially (e.g., regions with heat values ​​greater than the green threshold, yellow threshold, red threshold, and dark red threshold, respectively).

[0083] Through the above classification, the original image is divided into pixel blocks corresponding to different contribution levels. Each pixel block represents the contribution strength of that region to the correct identification of the current image into the corresponding category.

[0084] 6. Calculation of key feature block probability values Pixel blocks of different levels are processed, retaining pixels in the corresponding level while masking other areas. The processed image is then input into a pre-trained binary classification model for forward inference calculation, yielding the model's output confidence score (Conf). This confidence score (Conf) is used as the probability value for the corresponding pixel block. This probability value quantifies the degree of certainty regarding the classification result for an image region at a specific contribution level, and is used for subsequent image anomaly detection.

[0085] Through the above steps, multiple binary classification models can be obtained. Each binary classification model can identify an abnormal category of image and obtain the probability of each original sample image, thus obtaining the first probability distribution.

[0086] Through the above steps, this application can not only accurately identify multiple anomaly categories, but also locate key image regions that lead to model judgments through pixel-level contribution analysis, thereby improving the interpretability of the model and its adaptability to complex scenes.

[0087] The hybrid enhancement module 32 is used to perform hybrid enhancement on the original sample image to obtain a hybrid sample image, wherein the hybrid sample image has a second probability distribution representing its anomaly category, and the second probability distribution is associated with the first probability distribution; To further improve the robustness and generalization performance of the model in complex scenarios, this application introduces a hybrid enhancement strategy and a batch-level relational inference mechanism during the model training phase. The specific implementation process is as follows: Based on the obtained feature contribution heatmap and key feature blocks, regions with contribution values ​​higher than a preset threshold are selected as "high contribution mixed blocks".

[0088] When constructing training batches, for the current input image of the target category, high-contribution feature blocks are extracted from images of other categories and blended with the feature regions of the current image. This generates hybrid sample images containing multi-category semantic information. This process differs from traditional simple image stitching or parallel processing, aiming to simulate the complex situation of abnormal features intertwined with normal backgrounds in real-world scenes.

[0089] In the hybrid enhancement process, a batch-level graph neural network (GNN) module is introduced. Each sample in the current batch is treated as a node in a graph, and the graph neural network aggregates the relational features between samples. Through the message passing mechanism of the GNN, the potential associations between samples are inferred, thereby guiding the hybrid weights and position selection of feature blocks. This allows the model to not only focus on the features of a single image, but also to use the global relationships between samples within a batch for inference, enhancing the discriminative power of the feature representation.

[0090] Instead of using a traditional single hard label for the generated mixed sample images, new vector labels are dynamically generated based on the probability values ​​of image pixel blocks. .

[0091] Specifically, The dimension of the vector is consistent with the number of anomaly categories, and its value is calculated by weighting the proportion of other category feature blocks mixed into the current image and their corresponding confidence probability values. This vector label can more accurately reflect the distribution of each category feature in the mixed image and serve as a supervisory signal to guide model learning.

[0092] The advantage of this approach is that it allows the model to see more "mixed sample images" during training. For example, it stitches together the regions of Class A outlier images (red heatmap) and the regions of Class B outlier images (yellow heatmap). This allows the model to learn to judge the features of "mixed sample images," thereby improving the model's robustness and accuracy.

[0093] Training module 34 is used to input the mixed sample image and the original sample image as training samples into the model for training, and output a prediction classification result and a prediction sample identifier based on the first probability distribution and the second probability distribution. The prediction classification result is used to represent the probability distribution of the training sample belonging to the abnormal category, and the prediction sample identifier is used to identify the probability of the training sample belonging to the original sample image. A multi-classification model is trained by inputting mixed sample images and original sample images as training samples. The model outputs a predicted classification result and a predicted sample identifier based on a first probability distribution (i.e., the probability vector of the abnormal category to which the sample belongs) and a second probability distribution (i.e., the probability scalar of the sample being the original sample). The predicted classification result is the probability distribution vector of the training sample belonging to K abnormal categories, and the predicted sample identifier is the probability value of the training sample belonging to the original sample (1 represents the original sample, and 0 represents the mixed augmented sample).

[0094] The probability distribution calculation module 36 is used to obtain the final anomaly category probability distribution based on the predicted classification result and the predicted sample identifier, wherein the final anomaly category probability distribution is used to represent the probability that the test image of the input model belongs to any anomaly category.

[0095] The final anomaly category probability distribution is obtained through a dual-head output structure based on the predicted classification results and predicted sample identifiers. The final anomaly category probability distribution is generated as follows: the predicted classification results and predicted sample identifiers are multiplied by a temperature scaling parameter T, where the temperature scaling parameter T serves as a confidence gating mechanism. When the predicted sample identifier value is small (indicating that the sample image is a mixed sample image), the probability values ​​of all anomaly categories are suppressed, thereby realizing the calculation of the anomaly category probability distribution in the open set scenario. This distribution is used to characterize the probability that the test image of the input model belongs to any anomaly category.

[0096] The final probability distribution of the anomaly categories is as follows:

[0097] Where T is the temperature scaling parameter, Indicates the predicted classification result. Indicates the predicted sample identifier, As a confidence gating factor, when When the value is small, it indicates that the training samples are mixed sample images; when When the value is large, the predicted classification results of the training samples are retained.

[0098] This application's embodiments increase the diversity of anomaly categories learned by the model by using both original and mixed sample images during the training of a multi-input multi-output image detection model, thus enhancing the model's robustness. Simultaneously, the dual-head output forces the ResNet backbone network to learn more robust features by judging whether an image is "original" or "mixed" during training. These features participate in backpropagation, helping the model training to focus more on the classification of real images. Because of the probability distribution of mixed samples and training samples belonging to anomaly categories, using the trained model for image detection can overcome the problem of inaccurate identification of unknown categories in existing technologies, improving the accuracy of image anomaly category recognition.

[0099] Optionally, after obtaining the final anomaly category probability distribution based on the predicted classification results and predicted sample identifiers, the method further includes: Construct a classification loss function, which calculates the final anomaly class probability distribution based on the Kullback-Leibler divergence. The probability distribution vector of the mixed sample images The differences in distribution between them;

[0100] Among them, the probability distribution vector of the mixed sample images The probability distribution is dynamically generated during image blending enhancement. Its value is determined based on the pixel overlap and class confidence of the blended sample images and is used to characterize the weight ratio of each abnormal class feature in the blended sample. When the loss value starts to fluctuate or falls below the training stopping threshold If the training stops, then training should be stopped.

[0101] By introducing dynamic generation As a supervisory signal, combined with the KL divergence loss function, it can effectively constrain the model to accurately model the probability distribution of anomaly categories in mixed samples. Meanwhile, the gating mechanism of the temperature scaling parameter T and... The joint optimization further enhances the robustness and generalization ability of the model in open set anomaly detection tasks.

[0102] The hybrid sample images in this application employ multiple hybrid enhancement schemes, increasing the complexity of the samples and further improving the effectiveness of the model. This application embodiment uses two methods to perform hybrid enhancement on the original sample images.

[0103] (1) Generation based on probability mixup The process of blending and enhancing the original sample images to obtain a blended sample image includes: extracting a first sample image and a second sample image from the original sample images; setting the first sample image as the base image; extracting target pixel blocks from the second sample image as an overlay layer; setting the pixels of the first sample image to a first transparency parameter X at the pixel positions corresponding to the target pixel blocks, and setting the target pixel blocks of the second sample image to a second transparency parameter (1-X); and overlaying the first sample image and the second sample image according to the first transparency parameter X and the second transparency parameter (1-X) to generate a blended sample image.

[0104] Outputting the predicted classification result and predicted sample label based on the first probability distribution and the second probability distribution includes: calculating the degree of overlap between the target pixel block and the preset feature region in the first sample image. The preset feature region is the region in the first sample image where the pixel value is not the background color; based on the degree of overlap... The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Generate the probability distribution vector of the mixed sample images As the predicted classification result; set the predicted sample label for the mixed sample image, and mark the original sample probability as a preset value to indicate that the current sample is a mixed sample image generated by the mixed enhancement.

[0105] A first sample image and a second sample image are selected from the original sample image set as the mixing base. The first sample image is set as the base image, and the second sample image is used to extract the regions whose pixel values ​​are not the background color through an object detection algorithm to form target pixel blocks as the overlay layer.

[0106] At the pixel location corresponding to the target pixel block, a first transparency parameter X is set for the pixels in the first sample image, and a second transparency parameter (1-X) is set for the target pixel block in the second sample image, where X∈(0,1). The transparency parameter can be set based on a preset linear decay function or a random sampling strategy to control the weight ratio of the original sample features and the superimposed sample features in the mixed sample. The first sample image and the second sample image are pixel-level superimposed according to the first transparency parameter X and the second transparency parameter (1-X) to generate a mixed sample image. This mixed sample image simultaneously retains the background features of the base image and the target features of the superimposed layer, forming a semantically related composite image sample.

[0107] Based on pixel overlap The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Dynamically generate probability distribution vectors of mixed sample images . Specifically, The generation process includes: according to Calculate the feature weight ratio between the base image and the overlay layer. = × X and = (1- ) ×(1-X); The probability distribution of anomaly categories in the base map Anomaly category probability distribution of overlay layers According to weight and By performing weighted fusion, we obtain = × + × ; The weighted probability distribution is normalized to ensure... The numerical range of is [0,1]. A predicted sample identifier is set for the mixed sample image. The original sample probability is marked with a preset value S=0 to indicate that the current sample is a mixed sample image generated by bluffing enhancement; for the original sample image, the predicted sample identifier is set to S=1. This identifier is used to distinguish sample types during subsequent training to optimize the model output in conjunction with the gating mechanism of the temperature scaling parameter T.

[0108] In this embodiment of the application, pixel overlap is introduced. The joint calculation with the transparency parameter X, combined with the category confidence score. Dynamically generated This approach can accurately characterize the feature weight ratios of each anomaly category in mixed samples. Furthermore, the synergistic effect of setting the predicted sample identifier S and the dual-head output structure significantly improves the model's ability to discriminate mixed augmented samples, solving the technical challenge of unbalanced sample distribution in open set anomaly detection.

[0109] (2) Generation based on probability cutmix The process of blending and enhancing the original sample images to obtain blended sample images involves: extracting a first sample image and a second sample image from the original sample images; setting the first sample image as the base image; cropping target pixel blocks from the second sample image; and directly overwriting and pasting the target pixel blocks onto the corresponding positions of the first sample image to obtain the blended sample image.

[0110] Outputting the predicted classification result and predicted sample label based on the first probability distribution and the second probability distribution includes: identifying the preset feature region in the first sample image and calculating the degree of overlap between the target pixel block and the preset feature region. Based on the degree of overlap And the category confidence of the target pixel block in the anomaly category of the second sample image. Probability distribution vector for generating mixed sample images This serves as the predicted classification result. A predicted sample identifier is set for the mixed sample image, and the original sample probability is marked with a preset value to indicate that the current sample is a mixed sample image generated by the mixture enhancement.

[0111] The first and second sample images are selected from the original sample image set as the mixing base. The first sample image is set as the base image, and the second sample image is used to extract regions whose pixel values ​​are not background colors using an object detection algorithm, forming target pixel blocks as the overlay layer. These target pixel blocks are directly overlaid onto the corresponding positions in the first sample image to generate the mixed sample image. This overlay operation is achieved through coordinate mapping to ensure that the target pixel blocks are aligned with the semantic regions of the base image, thereby generating a semantically coherent composite image sample.

[0112] Identify a preset feature region in the first sample image. The preset feature region is a region in the first sample image whose pixel value is not the background color; its boundary can be determined using a semantic segmentation algorithm or an edge detection algorithm. Calculate the pixel overlap between the target pixel block and the preset feature region. .

[0113] Based on overlap And the category confidence of the target pixel block in the anomaly category of the second sample image. Dynamically generate probability distribution vectors of mixed sample images . Specifically, The generation process includes: according to Calculate the coverage weight of the target pixel block on the base map features. = × ; The probability distribution of anomaly categories in the base map Anomaly category probability distribution of target pixel block According to weight By performing weighted fusion, we obtain = (1 - ) × + × ; The weighted probability distribution is normalized to ensure... has a numerical range of [0, 1]. A prediction sample identifier is set for the mixed sample image. The original sample probability is marked as a preset value S=0, to indicate that the current sample is a mixed sample image generated by mixing enhancement; for the original sample image, the prediction sample identifier S is set to 1. This identifier is used to distinguish sample types in the subsequent training process, so as to cooperate with the gating mechanism of the temperature scaling parameter T to optimize the model output.

[0114] In the embodiments of the present application, by introducing the joint calculation of pixel coincidence degree and category confidence , combined with the semantic coverage strategy to dynamically generate , the feature weight proportion of each abnormal category in the mixed sample can be accurately represented. At the same time, the synergistic effect of the setting of the prediction sample identifier S and the dual-head output structure significantly improves the discrimination ability of the model for mixed enhanced samples, and solves the technical problem of unbalanced sample distribution in open set anomaly detection.

[0115] Optionally, in the actual use process of the multi-input multi-output model of the embodiments of the present application, it is first judged which abnormal category the image to be tested belongs to, and if it cannot be judged, the binary classification model is used for identification. Optionally, the method further includes: obtaining the maximum value of the final abnormal category probability distribution of the image to be tested ; when the maximum value is less than the open set determination threshold T1, it is considered that the image to be tested does not belong to any abnormal category; when the maximum value is greater than the category determination threshold T2 (T1<T2), it is considered that the image to be tested belongs to a definite abnormal category; when the maximum value is greater than the open set determination threshold T1 and less than the category determination threshold T2, the category to which the current image to be tested belongs cannot be determined, and the binary classification model is used for secondary identification, and the binary classification model is used to identify and output the category of the image to be tested.

[0116] In the case of fuzzy determination, a preset binary classification model is called to perform secondary identification on the image to be tested. The binary classification model is an independently trained auxiliary model, whose inputs are the feature vector of the image to be tested and the original abnormal category probability distribution , and the output is a binary classification result of "abnormal class" or "normal class". In the present invention, by introducing the synergistic mechanism of the multi-threshold determination strategy and the binary classification model, the fine classification of the open set anomaly detection task is realized. Wherein: Threshold determination logic: by setting the gradient thresholds of T1 and T2, the abnormal category probability distribution is divided into three decision intervals: "normal class", "definite abnormal class" and "fuzzy region", which improves the processing ability of the model for low-confidence samples; Synergistic effect of binary classification models: For samples in ambiguous regions, binary classification models introduce additional discrimination criteria (such as joint information of feature vectors and probability distributions) to compensate for the classification uncertainty of the main model in open set scenarios; Linkage with temperature scaling parameter T: The gating mechanism of temperature scaling parameter T forms a closed loop with the current judgment logic, ensuring the final probability distribution of anomaly categories. The confidence distribution conforms to the statistical characteristics of threshold determination.

[0117] The large model content management method based on hybrid enhancement provided in this application training a multi-input multi-output model, which improves the generalization ability of the model by diversifying the training samples. When using this model for image recognition, the abnormal category to which the image belongs can be given according to the output abnormal category probability. When the output result of the model is a decision on a fuzzy region, a binary classification model is introduced to further confirm the abnormal category to which the image belongs, thereby improving the recognition accuracy.

[0118] In practical applications, the large model content management method based on hybrid enhancement provided in this application can be applied to various electronic devices, such as personal computers, servers, mobile phones, and other devices with data processing capabilities. In one implementation, to improve processing efficiency, the large model content management method based on hybrid enhancement provided in this application can be applied to high-performance computers. Furthermore, the large model content management method based on hybrid enhancement provided in this application can be implemented through software, hardware, or a combination of both.

[0119] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0120] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0121] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0122] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0123] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0124] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in the above embodiments.

[0125] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape) or an optical medium (e.g., DVD).

[0126] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0128] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for managing large model content based on hybrid enhancement, characterized in that, The method includes: Receive raw sample images containing multiple anomaly categories, the raw sample images having a first probability distribution representing their respective anomaly categories; The original sample image is enhanced by blending to obtain a blended sample image, wherein the blended sample image has a second probability distribution representing its anomaly category, and the second probability distribution is associated with the first probability distribution; The mixed sample image and the original sample image are used as training samples and input into the model for training. Based on the first probability distribution and the second probability distribution, the model outputs a predicted classification result and a predicted sample identifier. The predicted classification result is used to represent the probability distribution of the training sample belonging to the abnormal category, and the predicted sample identifier is used to identify the probability of the training sample belonging to the original sample image. The final anomaly category probability distribution is obtained based on the predicted classification results and the predicted sample identifiers, wherein the final anomaly category probability distribution is used to represent the probability that the test image input to the model belongs to any anomaly category; Based on the first probability distribution and the second probability distribution, the output prediction classification result and prediction sample identifier include: Calculate the degree of overlap between the target pixel block and the preset feature region in the first sample image. The preset feature region is the region in the first sample image where the pixel value is not the background color. Based on the degree of overlap The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Generate the probability distribution vector of the mixed sample images. , as the predicted classification result; A predicted sample identifier is set for the mixed sample image, and the original sample probability is marked with a preset value to indicate that the current sample is a mixed sample image generated by fusion enhancement. The first sample image and the second sample image are extracted from the original sample image, and the target pixel block of the second sample image is used as an overlay layer, while the first sample image is used as a base image.

2. The method according to claim 1, characterized in that, The original sample images are blended and enhanced to obtain blended sample images, including: At the pixel position corresponding to the target pixel block, the pixel of the first sample image is set to the first transparency parameter X, and the target pixel block of the second sample image is set to the second transparency parameter (1-X); The first sample image and the second sample image are superimposed according to the first transparency parameter X and the second transparency parameter (1-X) to generate the mixed sample image.

3. The method according to claim 1, characterized in that, The original sample images are blended and enhanced to obtain blended sample images, including: Extract a first sample image and a second sample image from the original sample image, set the first sample image as the base image, and extract the target pixel block from the second sample image; The target pixel block is directly overlaid and pasted onto the corresponding position of the first sample image to obtain the mixed sample image.

4. The method according to claim 3, characterized in that, Based on the first probability distribution and the second probability distribution, the output prediction classification result and prediction sample identifier include: Identify a preset feature region in the first sample image and calculate the degree of overlap between the target pixel block and the preset feature region. ; Based on the degree of overlap And the category confidence of the target pixel block in the anomaly category of the second sample image. The probability distribution vector of the generated mixed sample images , as the predicted classification result; A predicted sample identifier is set for the mixed sample image, and the original sample probability is marked with a preset value to indicate that the current sample is a mixed sample image generated by the fusion enhancement.

5. The method according to claim 1, characterized in that, The final anomaly category probability distribution is as follows: Where T is the temperature scaling parameter, Indicates the predicted classification result. Indicates the predicted sample identifier, As a confidence gating factor, when When the value is small, it indicates that the training sample is a mixed sample image; when When the value is large, the predicted classification result of the training sample is retained.

6. The method according to claim 1, characterized in that, After obtaining the final anomaly category probability distribution based on the predicted classification results and the predicted sample identifiers, the method further includes: Construct a classification loss function, which calculates the final anomaly category probability distribution based on the Kullback-Leibler divergence. The probability distribution vector of the mixed sample images The differences in distribution between them; Wherein, the probability distribution vector of the mixed sample images The probability distribution is dynamically generated during image blending enhancement. Its value is determined based on the pixel overlap and class confidence of the blended sample images and is used to characterize the weight ratio of each abnormal class feature in the blended sample. When the loss value starts to fluctuate or falls below the training stopping threshold If the training stops, then training should be stopped.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the maximum value of the final anomaly category probability distribution of the image under test. ; When the maximum value If the value is less than the open set judgment threshold T1, the image under test is considered not to belong to any abnormal category. When the maximum value If the value is greater than the category determination threshold T2 (T1 < T2), then the image to be tested is considered to belong to a definite abnormal category; When the maximum value If the value is greater than the open set determination threshold T1 and less than the category determination threshold T2, the category of the current image to be tested cannot be determined. A binary classification model is then used for secondary recognition, which is used to identify and output the category of the image to be tested.

8. A large-model content management device based on hybrid enhancement, characterized in that, The device includes: A receiving module is used to receive raw sample images containing multiple anomaly categories, wherein the raw sample images have a first probability distribution representing the anomaly category to which they belong; A hybrid enhancement module is used to perform hybrid enhancement on the original sample image to obtain a hybrid sample image, wherein the hybrid sample image has a second probability distribution representing its anomaly category, and the second probability distribution is associated with the first probability distribution; The training module is used to input the mixed sample image and the original sample image as training samples into the model for training, and output a prediction classification result and a prediction sample identifier based on the first probability distribution and the second probability distribution. The prediction classification result is used to represent the probability distribution of the training sample belonging to the abnormal category, and the prediction sample identifier is used to identify the probability of the training sample belonging to the original sample image. The probability distribution calculation module is used to obtain the final anomaly category probability distribution based on the predicted classification result and the predicted sample identifier, wherein the final anomaly category probability distribution is used to represent the probability that the test image input to the model belongs to any anomaly category; The training module is also used for: Based on the first probability distribution and the second probability distribution, the output prediction classification result and prediction sample identifier include: Calculate the degree of overlap between the target pixel block and the preset feature region in the first sample image. Wherein, the preset feature region is the region in the first sample image whose pixel value is not the background color; based on the degree of overlap The first transparency parameter X and the category confidence of the anomaly category to which the second sample image belongs. Generate the probability distribution vector of the mixed sample images. As the predicted classification result; set a predicted sample identifier for the mixed sample image, and mark the original sample probability as a preset value to indicate that the current sample is a mixed sample image generated by the mixed enhancement, wherein the first sample image and the second sample image are extracted from the original sample image, and the target pixel block of the second sample image is used as an overlay layer, and the first sample image is used as a base image.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images