Deep neural network fuzzing test case generation method, system and application based on cross-modal generation and distribution perception

Abstract

This invention discloses a deep neural network-based fuzzy testing method and system based on cross-modal generation and distribution awareness, belonging to the fields of software testing and artificial intelligence security. Addressing the problems of insufficient semantic diversity in generated test cases, uneven data distribution, and poor adaptability to high-resolution image datasets in existing testing methods, this invention uses clustering to select seed samples and extract abstract text embedding vectors. It then generates prompt words using a scene library, adjusts the generation quantity for each category according to a distribution awareness strategy, filters invalid data to obtain a valid test set, and inputs it into the model to statistically analyze error samples. This invention improves the diversity and effectiveness of test cases, increases test coverage and the ability to detect defects in few sample categories, and enhances model robustness.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence security, deep learning software testing and data augmentation technology, and specifically relates to a method, system and application for generating deep neural network fuzzy test cases based on cross-modal generation and distribution perception. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep neural networks have demonstrated outstanding performance in tasks such as image classification, object detection, and natural language processing, and have been widely applied in security-critical fields such as autonomous driving, medical image diagnosis, and financial risk control. However, deep neural networks generally suffer from insufficient robustness, meaning that when faced with input data that differs slightly from their training data distribution or is previously unseen, they are highly prone to producing incorrect predictions. This uncertainty poses serious security risks to practical applications. Therefore, how to conduct thorough and effective testing of deep neural networks to evaluate and improve their reliability has become a focus of attention for both academia and industry.

[0003] To discover defects in deep neural network models, various testing techniques have been proposed, mainly categorized into coverage-guided testing and distribution-aware testing. Coverage-guided testing draws on traditional software testing principles, proposing metrics such as neuron coverage, neuron boundary coverage, and strong neuron activation coverage, aiming to guide test case generation by maximizing these metrics. Distribution-aware testing, on the other hand, focuses on the distribution characteristics of the generated data, attempting to generate test cases that conform to the statistical patterns of the training data, or exploring the distribution boundaries between different categories.

[0004] Although existing technologies have driven the development of deep neural network testing to some extent, significant technical defects and limitations still exist when facing increasingly complex real-world application scenarios and high-resolution datasets.

[0005] First, existing methods primarily rely on simple variations of the original image, making it difficult to generate high-quality semantic diversity. Existing testing tools typically utilize affine transformations (such as rotation, translation, and scaling) or pixel-level perturbations (such as adding Gaussian noise, blurring, and adjusting contrast) to generate new test samples. While these low-level variations can change the pixel values ​​of an image, they are strictly limited by the feature space of the original data and cannot introduce advanced semantic changes. For example, traditional methods cannot transform a "dog on the grass" image into a "dog on the snow" image through simple mathematical transformations, nor can they change the pose, style, or background environment of an object. This results in highly repetitive semantically generated test cases, lacking diversity and failing to simulate the complex and varied scenarios of the real world, thus limiting the sufficiency of the tests.

[0006] Second, existing methods typically assume that valid test inputs must strictly follow the distribution of the training data, making it difficult to address the problem of data scarcity. In practical applications, due to high data collection costs or privacy restrictions, training data often cannot cover all real-world scenarios. When generating test cases, existing methods tend to mutate based on the abundant "head" class data, while for the scarce "tail" classes (classes with few samples), it is difficult to generate valid test samples due to a lack of sufficient seed data and prior knowledge. This blind replication of the original imbalanced distribution leads to severe undertesting of the model on scarce classes, making it impossible to effectively evaluate and improve its performance in edge cases.

[0007] Third, their applicability to high-resolution and semantically complex datasets is poor. Early testing methods were mostly validated on low-resolution, semantically simple handwritten digit or small icon datasets. However, in high-resolution datasets commonly used in fields such as autonomous driving (e.g., ImageNet), images have rich details and complex semantic structures. In these high-dimensional feature spaces, simple pixel perturbations are often submerged in complex background textures, making it difficult to trigger deep logical errors in the model; or, excessive pixel perturbations can destroy the semantic structure of the image, causing the generated image to no longer belong to any category (i.e., invalid data), thus losing its testing significance. Although existing distribution-guided methods attempt to explore distribution boundaries, in high-dimensional spaces, the samples they generate often still cluster near the original data, failing to effectively explore the vast unknown valid input space.

[0008] In summary, existing deep neural network testing technologies, when generating test cases, are limited by the characteristics and distribution of the original training data and lack the ability to utilize external general knowledge. This results in insufficient diversity and semantic simplicity in the generated test data, and they are unable to effectively address the challenges posed by long-tail distribution and data scarcity.

[0009] Therefore, how to utilize external knowledge (such as the general visual concepts contained in large-scale pre-trained models) to generate test cases that are both consistent with real semantics and highly diverse, and that can balance data distribution, is a problem that current technology urgently needs to solve. Summary of the Invention

[0010] To address the shortcomings of existing technologies, this invention aims to provide a deep neural network-based fuzzy testing method based on cross-modal generation and distribution awareness. This method leverages the powerful generative capabilities and prior knowledge of artificial intelligence-generated content models (such as text-to-image models) to transform training data into abstract modalities, thereby generating effective test cases covering a wider distribution, with particularly targeted enhancements for classes with fewer samples.

[0011] The proposed method for generating fuzzy test cases based on cross-modal generation and distribution awareness using deep neural networks includes the following steps:

[0012] Step S1: Cluster the original training dataset of the target deep neural network and select seed samples from each cluster; initialize learnable text tags, use text inversion technology to invert the seed samples, learn abstract text embedding vectors representing the visual semantic features of the cluster, and bind them to the aforementioned learnable text tags.

[0013] Step S2: Based on the abstract text embedding vector, construct diverse text prompt words in combination with a preset scene description library; use a text-to-image model to generate new image test cases based on the text prompt words; during the generation process, adaptively adjust the number of test cases generated for each category according to the sample quantity distribution of each category in the original training data.

[0014] Step S3: Extract the feature vector of the generated image and calculate its feature distance from the cluster center of the original training data; filter out test cases whose feature distance exceeds the preset deviation threshold according to the preset deviation threshold to obtain the final effective test set.

[0015] In step S1 of this invention, the clustering process employs an adaptive algorithm selection mechanism. The clustering effects of the K-Means and DBSCAN clustering algorithms are evaluated by calculating silhouette coefficients, and the algorithm with the highest silhouette coefficient is selected to group the original training dataset. The silhouette coefficient is calculated as follows:

[0016] For any sample x, its silhouette coefficient ;

[0017] in, Let x be the average distance between sample x and other samples in the same cluster. Let x be the average distance between sample x and all samples in the nearest neighbor cluster.

[0018] In step S1 of this invention, the text inversion technique involves: freezing the encoder parameters of the text-to-image generation model and optimizing specific learnable labels for each cluster. The initial dimension of the learnable tag is a preset fixed-dimensional floating-point vector. The vector representation of the tag is updated through a preset optimization algorithm so that it can reconstruct the visual semantics of the seed sample to the greatest extent in the text embedding space.

[0019] In step S2 of this invention, the adaptive adjustment of the number of test cases generated for each category is calculated using the following distribution-aware formula. Number of classes generated :

[0020]

[0021] in, The preset total number of generators, The total number of categories, The set of training sample numbers for each category. For the first The original number of training samples for the class. This represents the maximum number of training samples across all categories.

[0022] In step S2 of this invention, the scene description library contains various real-world environment descriptions and is classified according to scene type or semantic dimension; the construction rule of the text prompt words is: embedding abstract text into learnable tags corresponding to vectors. Combined with description templates in the scene description library, The basic prompt word is "Photo in [Scene Description]"; for specific categories, add category constraints and generate a format of " The prompt word is "a photo with a [category name] in the [scene description]"; the scene description library supports dynamic expansion, and the expansion meets the semantic consistency verification rules.

[0023] In step S3 of this invention, the feature distance calculation method is as follows: input the generated test case into the pre-trained feature extraction model to obtain the feature vector of the previous layer of the output layer; calculate the Euclidean distance between the feature vector and all cluster centers of the original training data, and take the minimum value as the deviation of the test case; the feature extraction model is any one of ResNet, VGG, AlexNet or MobileNet.

[0024] In step S3 of this invention, the deviation threshold is a dynamic threshold derived from the statistical distribution of training data, which is the 95th or 90th percentile of the feature distance distribution of the training data; the threshold adjustment is triggered by changes in the dataset size or category distribution, and the training data distribution is re-statistically analyzed and the threshold is calculated during adjustment.

[0025] In this invention, when the original training data is large, step S3 employs an approximate deviation calculation method: first, it verifies whether the feature vectors of the training data follow a multivariate normal distribution; if so, it utilizes... The rule is to calculate the proportion of elements in the generated sample feature vector that exceed 3 times the standard deviation. When this proportion exceeds a preset threshold, it is judged as an invalid test case.

[0026] This invention also introduces an application of the above methods in evaluating model defects or retraining models. The effective test set is input into the target deep neural network, and samples whose statistical classification output is inconsistent with the expected label are used as error samples to evaluate model defects or retrain models.

[0027] Based on the above, this invention also proposes a deep neural network fuzzy test case generation system based on cross-modal generation and distribution awareness, comprising:

[0028] The abstract modality generation module is used to cluster the original training dataset, select seed samples from each cluster, learn abstract text embedding vectors representing the visual semantic features of the cluster through text inversion technology, and bind them to learnable text tags.

[0029] The test case generation module is used to construct diverse text prompts based on abstract text embedding vectors combined with a preset scene description library, generate new image test cases using a text-to-image model, and adaptively adjust the number of test cases generated for each category according to the sample quantity distribution of each category in the original training data.

[0030] The data cleaning module is used to extract the feature vectors of the generated images, calculate the feature distance between them and the cluster centers of the original training data, filter invalid test cases according to the preset deviation threshold, and obtain the valid test set.

[0031] In this invention, the abstract modality generation module includes:

[0032] Clustering units employ either the K-Means clustering algorithm or the DBSCAN clustering algorithm, selecting the optimal clustering algorithm by calculating the silhouette coefficient to group the original training dataset.

[0033] The seed selection unit selects the most representative seed samples from each cluster according to a preset number or proportion. The selection criteria are quantitative indicators that combine intra-cluster distance and center distance.

[0034] The text inversion learning unit is used to initialize learnable text tags, freeze the encoder parameters of the text-to-image model, and update the vector representation of the learnable text tags through an optimization algorithm.

[0035] In this invention, the test case generation module includes:

[0036] The scene library unit is used to store descriptions of real-world environments categorized by scene type or semantic dimension, and supports dynamic expansion and semantic consistency verification.

[0037] The text generation unit is used to combine learnable text tags with description templates in the scene library to generate diverse text prompts;

[0038] The distribution-aware control unit is used to statistically analyze the sample quantity distribution of each category in the original training data and calculate the number of test cases generated for each category using the distribution-aware formula described in claim 5.

[0039] In this invention, the data cleaning module includes:

[0040] The feature extraction unit is used to call a pre-trained feature extraction model to extract feature vectors from the generated image.

[0041] The similarity calculation unit is used to calculate the Euclidean distance between the generated image feature vector and the cluster centers of the original training data, and to obtain the deviation using an approximate deviation calculation method.

[0042] The data filtering unit is used to filter invalid test cases based on a dynamic deviation threshold to obtain a valid test set.

[0043] In this invention, the interaction between the abstract modality generation module, test case generation module, data cleaning module, and model evaluation module is either serial or parallel, and the data transmission format adopts a preset standardized vector format; the system supports distributed deployment, and a load balancing scheduling mechanism is used when multiple GPUs generate test cases in parallel.

[0044] This invention also proposes a deep neural network fuzzy measurement system based on cross-modal generation and distribution sensing, comprising:

[0045] Based on the above-described deep neural network fuzzy test case generation system based on cross-modal generation and distribution awareness;

[0046] The model evaluation module is used to input the valid test set into the target deep neural network, statistically classify the erroneous samples that are inconsistent with the expected labels, and use them to evaluate model defects or perform model retraining.

[0047] The beneficial effects of the deep neural network-based fuzzy test case generation method and system based on cross-modal generation and distribution awareness proposed in this invention include:

[0048] 1. Significantly Improved Test Case Diversity and Effectiveness: This invention utilizes cross-modal generation technology to introduce background, lighting, texture, and style variations not present in the original data, thereby overcoming the limitations of traditional pixel-level mutations (such as rotation and noise addition). Experimental data shows that the test cases generated by this invention perform exceptionally well in maintaining semantic effectiveness, significantly outperforming traditional methods based on adversarial attacks or genetic algorithms. This means that the generated images are not only diverse but also more consistent with the visual semantic distribution of the real world.

[0049] 2. Significant Breakthrough in Test Coverage: Experimental results show that the test set generated by this invention can reach the decision logic boundaries of deep neural networks more deeply. In key coverage metrics, this invention outperforms existing state-of-the-art technologies. This demonstrates that this invention can more comprehensively evaluate the model state.

[0050] 3. Strong Error Detection Capability: Thanks to the distribution-aware strategy, this invention can accurately locate the model's weaknesses in few-sample classes. Experimental statistics show that, within the same time frame, this invention detects far more unique errors than other methods. Especially in few-sample classes, the number of errors detected by this invention is orders of magnitude higher, effectively exposing potential reliability issues in data-scarce scenarios. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the overall system architecture and workflow of the method of the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the principle of the abstract modality generation module in this invention.

[0054] Figure 3 This is a comparison of specific images of test cases generated in this embodiment of the invention with test cases generated by other industry methods. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0056] This invention formally defines the data generation model and the test case optimization problem with distribution-aware constraints in the long-tailed distribution scenario of deep neural network testing, providing a clear mathematical foundation for algorithm design; it proposes a cross-modal fuzzy testing method for high-resolution image classification models, which significantly improves the diversity and effectiveness of test data by mapping images to abstract text modalities, utilizing the prior knowledge of generative models, and combining distribution-aware strategies and cluster-based cleaning strategies.

[0057] The deep neural network fuzz testing method based on cross-modal generation and distribution awareness proposed in this invention includes the following steps:

[0058] Step S1: Abstract Modality Extraction Step. The original training dataset of the target deep neural network is clustered, and seed samples are selected from each cluster. Learnable text tags are initialized, and text inversion technology is used to invert the seed samples to learn abstract text embedding vectors representing the visual semantic features of the cluster, which are then bound to the aforementioned learnable text tags.

[0059] Step S2: Distribution-Aware Test Case Generation Step. Based on the abstract text embedding vector, diverse text prompts are constructed using a pre-defined scene description library; new image test cases are generated using a text-to-image model based on the text prompts; during the generation process, the number of test cases generated for each category is adaptively adjusted according to the sample quantity distribution of each category in the original training data.

[0060] Step S3: Data cleaning step. Extract the feature vectors of the generated images and calculate their feature distances to the cluster centers of the original training data; based on a preset deviation threshold, filter out test cases whose feature distances exceed the threshold to obtain the final valid test set;

[0061] Step S4: Fuzz Testing and Evaluation. Input the valid test set into the target deep neural network. Collect samples whose classification output does not match the expected label as error samples, which are used to evaluate model defects or for model retraining.

[0062] In step S1, the abstract modality extraction technique employs a clustering method and a text inversion method for the samples. The clustering method extracts representative samples from the original samples using KMeans or DBSCAN clustering algorithms. The text inversion method freezes the encoder of the text-to-image model and optimizes a specific learnable label for each cluster. This enables the reconstruction of the visual concept of the seed sample in the text embedding space.

[0063] In step S2, the adaptive adjustment of the number of test cases generated for each category is specifically calculated using the following distribution-aware formula. Number of classes generated :

[0064]

[0065] in, The preset total number of generators, The total number of categories, The set of training sample numbers for each category. For the first The original number of training samples for the class. This represents the maximum number of training samples across all categories.

[0066] In step S2, constructing diverse text prompts by combining a preset scene description library includes: maintaining a scene library containing various real-world environment descriptions, specifically a set of template strings containing various real-world environment descriptions, such as "a picture of __ on a snow-capped mountain"; and using the abstract tags obtained in step S1. Combined with description templates from the scene library, the generated format is "a sheet of paper". The prompt word is "Photo in [Scene Description]".

[0067] In step S3, the calculation of feature distance specifically involves: inputting the generated test case into the pre-trained feature extraction model to obtain feature vectors, calculating the Euclidean distance between the feature vector and all cluster centers of the training data, and taking the minimum value as the deviation of the test case;

[0068] The filtering specifically involves determining that, for all generated test cases, if the deviation is greater than 90% or more compared to other test cases of the same category, the test case is invalid.

[0069] When the training data size is not less than 50,000 images (such as the CIFAR10 dataset), an approximate deviation calculation method is used: assuming the training data feature vectors follow a multivariate normal distribution, and utilizing... The rule is to calculate the proportion of elements in the generated sample feature vector that exceed 3 times the standard deviation, and use this as the approximate deviation.

[0070] This invention relates to a deep neural network-based fuzzy test case generation system based on cross-modal generation and distribution awareness, comprising:

[0071] The abstract modality generation module is used to perform data clustering and inversion learning of abstract text vectors based on representative data;

[0072] The test case generation module is used to generate image data based on the distribution-aware strategy and the scene library;

[0073] The data cleaning module is used to calculate feature distances and filter invalid data;

[0074] The model evaluation module is used to perform fuzz testing and output a set of error samples.

[0075] Specifically, the new testing framework of this invention comprises three core components: an abstract modality generator, a test case generator, and a data cleaner.

[0076] 1. Abstract Modal Generator

[0077] The main function of this component is to transform high-dimensional raw image data into low-dimensional, highly generalized abstract text representations. The specific implementation steps are as follows:

[0078] Clustering and Adaptive Algorithm Selection: To extract representative features from a large amount of redundant training data, the training dataset is first clustered. Since different datasets have different data distribution characteristics, this invention is not limited to a single clustering algorithm but introduces an adaptive selection mechanism. The system simultaneously evaluates the performance of the K-Means clustering algorithm and density-based clustering algorithms on the current data, measuring the clustering effect by calculating the silhouette coefficient. The silhouette coefficient combines intra-cluster compactness and inter-cluster separation. The system automatically selects the algorithm with the higher silhouette coefficient to group the current data, thereby ensuring that samples with similar characteristics (such as similar backgrounds, angles, or styles) are grouped into the same cluster.

[0079] Seed sample selection: In order to improve the efficiency of subsequent calculations and avoid overfitting, the system does not directly use all the images within a cluster, but randomly selects 3 images that are closest to the cluster center from each cluster as "seed samples".

[0080] Text inversion learning: Utilizing text inversion techniques for cross-modal mapping. The system keeps the encoder parameters of the pre-trained text-to-image generation model unchanged, and initializes a specific learnable label (tag) for each cluster, denoted as . By employing gradient-based optimization algorithms (such as Adam optimization) with the objective function of minimizing the noise estimation loss of the diffusion model, the vector representation of the tag is iteratively updated during backpropagation, enabling it to reconstruct the visual semantics of the seed samples to the greatest extent possible within the text embedding space. Ultimately, each cluster is transformed into an abstract tag that captures the high-level semantic concepts of the image group (such as the core shape of an object or specific texture), while ignoring task-irrelevant pixel-level details.

[0081] 2. Distribution-Aware Test Case Generator

[0082] This component is responsible for generating diverse and evenly distributed test cases using abstract tags. The specific implementation steps are as follows:

[0083] Diverse prompt word construction: The system maintains a scene description library containing various real-world environment descriptions predefined by requirements engineers (e.g., "in the snow," "in the rain," "oil painting style," etc.). The system will then use the abstract tags obtained in step 1... By arranging and combining descriptions from the scene library, a scene such as "a The text prompt "Photos in [Scene]". For specific categories, to prevent semantic drift, a category constraint is explicitly added to the prompt, formatted as " ", a [category name]"

[0084] Distribution-Aware Generation Strategy: To address the long-tail distribution problem in training data, this invention introduces a dynamic weight generation and calculation mechanism. The system counts the number of samples of each category in the training set and adaptively calculates the weights according to the following formula. Number of test cases generated for the class :

[0085]

[0086] in, This represents the preset target total number of generated. Represents the total number of categories. It is the set of training sample counts for all categories. It is the first The original number of training samples for the class. It is the maximum number of samples across all categories.

[0087] This formula ensures that the fewer the original training samples of a category, the more generation quota is obtained, thereby achieving "oversampling" enhancement for fewer sample categories and balancing the category distribution of the test data.

[0088] 3. Cluster-based data cleaner

[0089] This component is used to ensure the validity of the generated data and remove images with semantic errors or excessive deviations. The specific implementation steps are as follows:

[0090] Feature extraction and distance calculation: The generated test cases are input into the pre-trained feature extraction model (taking the vector from the layer preceding its output layer) to obtain its feature vectors. Simultaneously, the center vectors of each cluster in the original training data are pre-calculated. The system calculates the Euclidean distance between the feature vector of the generated image and the centers of all training data clusters, and takes the minimum value as the deviation of the image.

[0091] Dynamic threshold filtering: To accommodate the intra-class variability, the system does not use a fixed threshold, but instead calculates a dynamic threshold based on the distribution of the training data. If the deviation of a generated image exceeds the threshold, the image is determined to have deviated excessively from the semantic space of the target class (i.e., invalid data) and is removed from the test set.

[0092] Large-scale data optimization: To address the extremely large scale of training data and reduce the overhead of calculating Euclidean distance, this invention also designs an approximate deviation calculation method. Assuming the feature vectors of the training data follow a multivariate normal distribution, this method utilizes... The principle is to statistically analyze the proportion of elements in the generated image feature vectors that exceed three standard deviations from the mean of the training data. A higher proportion indicates a greater deviation. This approximate calculation can significantly improve the efficiency of cleaning large-scale datasets.

[0093] The fuzz testing system of this invention includes an abstract modality generation module, a test case generation module, and a data cleaning module;

[0094] Figure 1 The diagram shows the overall architecture and module structure of the fuzzy testing system of the present invention. The system constructs an abstract semantic space based on the input raw training data and outputs a high-quality test case set. Figure 2 This is a schematic diagram of the principle of the abstract modality generation module in the test system of the present invention. The module learns learnable tags in the frozen generative model based on representative samples output by clustering units, using text inversion technology to capture the high-level semantic features of the image.

[0095] The abstract modality generation module further includes: a clustering unit as the core of data preprocessing, a seed selection unit for screening representative samples, and a text inversion learning unit for learning the abstract semantics of images.

[0096] The test case generation module further includes: a scene library unit for storing environment and style descriptions, a text generation unit for constructing diverse prompt words, and a distribution-aware control unit for statistically analyzing category distribution and controlling the number of generated words;

[0097] The data cleaning module further includes: a feature extraction unit for extracting high-dimensional features of the image, a similarity calculation unit for calculating the spatial distance of features, and a data filtering unit for removing invalid data.

[0098] like Figure 1 As shown, the fuzz testing method of the present invention includes the following steps:

[0099] Step 1: Data Preprocessing and Abstraction

[0100] For the input training dataset, it is first grouped by clustering units. The system calculates silhouette coefficients to automatically select the optimal clustering algorithm (such as K-Means or DBSCAN). Within each cluster, seed selection units are randomly selected. Each seed sample is used as a seed. Subsequently, the text inversion learning unit uses a pre-trained model to map these seed samples into abstract tokens in the text embedding space, completing the transformation from pixel modality to abstract semantic modality.

[0101] Step 2: Setting the Distributed Sensing Strategy

[0102] Before generating test cases, the distribution-aware control unit counts the number of samples in each category of the training data. If a long-tailed distribution is detected (i.e., a category with scarce samples exists), the number of test cases to be generated for each category is calculated according to a preset distribution balancing formula. This strategy ensures that categories with fewer samples receive higher generation weights, thereby covering distribution areas ignored by the original data.

[0103] Step 3: Generate Diverse Use Cases

[0104] The text generation unit combines environmental descriptions (such as "snowy" and "night") from the scene library unit with the abstract tags obtained in step one to construct diverse text prompts. Subsequently, the text-to-image model is invoked to generate a large number of candidate image test cases in parallel, based on the quantity instructions calculated in step two.

[0105] Step 4: Data Validation Cleaning

[0106] The feature extraction unit uses a pre-trained classification model to extract feature vectors from the generated image. The similarity calculation unit calculates the Euclidean distance between this vector and the cluster centers of the original training data. The data filtering unit determines whether the image deviates excessively from the target semantics based on a dynamic threshold or the 3-sigma principle. If the deviation is too large, pruning (filtering) is performed; if retained, it is added to the final set as a valid test case.

[0107] Step 5: Fuzzy Testing and Evaluation

[0108] The cleaned test cases are input into the deep neural network model under test. Samples whose statistical model predictions do not match the expected labels are marked as erroneous samples and used for subsequent model robustness evaluation or retraining correction.

[0109] The specific algorithm of the scheduling method of this invention is as follows:

[0110]

[0111] Original training data Typically, data contains a large number of redundant samples (such as multiple photos of the same object taken from similar angles). Directly using all the data for computation is inefficient and prone to overfitting. This invention first performs a clustering operation. In this step, the system is not limited to a single clustering algorithm but calculates silhouette coefficients to evaluate clustering quality. The system automatically compares the silhouette coefficients of the K-Means and DBSCAN algorithms on the current data, selecting the algorithm with the higher coefficient value to divide the training data into several clusters. Subsequently, to reduce the computational cost of subsequent text inversion, the system performs a seed selection operation. For each cluster, instead of using all images within the cluster, the system randomly selects three images closest to the cluster center as "seed samples" and stores them in... The set is used for subsequent generation of abstract modal data.

[0112] System traversal of seed set For each seed sample, a pre-trained generative model is used for text inversion learning. Specifically, the system fixes the parameters of the generative model and optimizes only a randomly initialized label vector to reconstruct the visual features of the seed sample to the greatest extent possible in the text embedding space. Ultimately, each cluster is mapped to a unique abstract label (denoted as ). These tags are collected in a set This is used for subsequent generation tasks.

[0113] Upon entering the generation task, the system first executes a distribution-aware strategy. Existing technologies often ignore the distribution of data, leading to insufficient testing for few-sample classes. This system counts the number of samples for each class in the training data and calculates the generation weights according to a specific formula. This strategy ensures that classes with fewer training samples receive a larger generation quota. Subsequently, the system invokes the text generation unit. This unit abstracts the tags... With scene library The image generation unit combines environmental descriptions (such as "in the rain" and "sketch style") to construct diverse cue words. Then, using these cue words, the image generation unit generates a large number of candidate test cases in parallel according to calculated weights, and stores them in... This set is used for subsequent testing tasks.

[0114] The generated images may have semantic drift or quality issues, therefore cleaning is required. System calculations. The system calculates the feature distance (Euclidean distance) between each generated test case in the set and the cluster centers of the original training data. If the feature distance of an image exceeds a dynamic threshold derived from the statistical distribution of the training data (e.g., the cutoff point of the top 30% after distance sorting), the image is deemed to have deviated excessively from the target semantics and is considered invalid data. The system then performs a pruning operation, removing the image from the test case set. Remove from the middle. For large-scale datasets, the system uses a method based on... The principle of approximate calculation method accelerates distance determination.

[0115] After obtaining a cleaned, high-quality test case set, the system inputs it into the deep neural network model under test. The system compares the model's predicted output with the pre-defined labels of the test cases one by one. If they do not match, the model is determined to have a defect on this sample, and the sample is added to the error set. Finally, the algorithm outputs all detected erroneous samples, which can be further used for model retraining and repair, thereby improving the model's robustness in long-tailed distribution scenarios.

[0116] The distributed-aware fuzzy testing problem in this invention involves key data structure definitions, generation strategy formulas, and data validity constraints, specifically including:

[0117] (1) Clustering quality assessment constraints

[0118] To extract the most representative features from large-scale training data, the algorithm introduces the silhouette coefficient as an evaluation metric for clustering quality. For any sample... Its profile coefficient Defined as:

[0119]

[0120] in, Indicates sample The average distance from other samples within the same cluster (cluster compactness). Indicates sample The average distance to all samples in the nearest neighbor cluster (inter-cluster separation). During the initialization phase, the system automatically calculates the average silhouette coefficients generated by different clustering algorithms and selects the clustering results with higher coefficients to construct the cluster set. The seed selection unit only extracts samples from the optimized clusters to ensure that the subsequently generated abstract labels can accurately represent the visual semantic core of the cluster.

[0121] (2) Constraints on the number of distributed sensing generation

[0122] To address the long-tail distribution problem in training data and ensure that generated test cases cover categories with scarce samples, this invention defines distribution-aware generation constraints. Let... The set of sample sizes for each class in the training set. The largest sample size. Generate a total number for the preset target. The number of categories. For any number of categories. The number of its generation The following constraint formula must be satisfied:

[0123]

[0124] This formula indicates the number of original training samples for each category. The fewer the number, the less generation quota is allocated to it. The more. This constraint forces the generator to tilt towards the "tail" category, thus affecting the final test case set. This achieves a balanced distribution of categories and corrects model bias.

[0125] (3) Data semantic validity and cleaning constraints

[0126] To ensure that the generated test cases semantically belong to the target category and do not deviate excessively from the real data distribution, this invention defines a distance constraint based on feature space. For the generated test cases... The feature vector is extracted using a pre-trained feature extractor. Calculated in Among the cluster centers, the nearest cluster center is... Euclidean distance as deviation :

[0127]

[0128] Algorithm sets dynamic threshold (e.g., the distance distribution within the training data) quantiles). If If a sample violates the validity constraint, it is considered invalid noise and must be removed during the cleaning phase. For large-scale datasets, the system further introduces an approximate constraint: assuming the feature vectors follow a multivariate normal distribution, if more than a certain proportion of the elements in the feature vectors fall within the mean of the training data... If the data is outside the (standard deviation) range, it is also considered invalid.

[0129] Example

[0130] The method described in this invention focuses on the reliability testing and enhancement of deep neural networks in scenarios with long-tailed distributions and scarce data. To verify the effectiveness of this invention, this embodiment constructs a public dataset with typical long-tailed distribution characteristics and mainstream deep learning models from the industry for experimental verification. The specific process is as follows:

[0131] 1. Experimental Scenario and Subjects: This embodiment selects the Food-101 and ImageNet100 datasets as experimental subjects. Both datasets are resampled to obtain long-tailed versions, rigorously simulating the extremely imbalanced class scenarios in the real world. The deep neural network model to be tested is ResNet50, a widely used benchmark model in image classification tasks.

[0132] 2. Hardware and Software Environment: The automated testing system code in this embodiment is deployed on a high-performance computing workstation. The hardware configuration includes an Intel Xeon Platinum 8336C CPU, 128GB of RAM, and four NVIDIA RTX 4090 GPUs. The software environment is an Ubuntu 20.04 operating system, using the PyTorch 2.2.1 framework for training and inference of deep learning models, and employing Stable Diffusion V2 as the base model for text-to-image generation.

[0133] 3. Visual quality assessment of generated images: To accurately assess the semantic quality of the generated data, Figure 3 The example demonstrates a comparison of specific generated image samples. This embodiment uses the "sweet cake" category for illustration. Figure 3 As shown, traditional adversarial attack methods (such as BIM and PGD) only add noise at the pixel level, invisible to the naked eye, resulting in simple visual semantics; existing mutation methods and distribution-guided methods (such as SENSEI and DistXplore) generate images with repetitive backgrounds and textures. In contrast, the images generated by this invention ( Figure 3 The rightmost column contains rich background variations and object pose changes, with the main objects clearly identifiable. This demonstrates that the present invention, by introducing a scene library and abstract markers, significantly improves the visual diversity of test cases while maintaining semantic validity.

[0134] 4. Test Coverage Evaluation: To quantify the triggering degree of the test cases generated by this invention on the deep neural network logic, this embodiment uses four industry-standard neuron coverage metrics for evaluation: k-multi-segment neuron coverage (KMNC), neuron boundary coverage (NBC), strong neuron activation coverage (SNAC), and Top-K neuron coverage (TKNC). This invention outperforms existing adversarial attack methods (BIM, PGD) and distribution-oriented methods (SENSEI, DistXplore) on most coverage metrics. This indicates that the test cases generated by this invention using cross-modal generation technology can explore the extreme regions of the model's decision boundary (i.e., regions with extremely high or low activation values) more deeply, thus testing the model's neuronal logic more thoroughly than traditional pixel mutation methods.

[0135] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the interpreted scripting language Python.

[0136] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0140] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations. The scope of protection of this invention is not limited to the above embodiments. Variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A method for generating fuzzy test cases based on cross-modal generation and distribution awareness using deep neural networks, characterized in that, Includes the following steps: Step S1: Perform clustering on the original training dataset of the target deep neural network and select seed samples from each cluster; Initialize learnable text tags, use text inversion technology to invert the seed samples, learn abstract text embedding vectors representing the visual semantic features of the cluster, and bind them to the aforementioned learnable text tags; Step S2: Based on the abstract text embedding vector, construct diverse text prompt words in combination with a preset scene description library; Use a text-to-image model to generate new image test cases based on the text prompts; During the generation process, the number of test cases generated for each category is adaptively adjusted based on the sample quantity distribution of each category in the original training data. Step S3: Extract the feature vector of the generated image and calculate its feature distance from the cluster center of the original training data; filter out test cases whose feature distance exceeds the preset deviation threshold according to the preset deviation threshold to obtain the final effective test set.

2. The method according to claim 1, characterized in that, In step S1, the clustering process employs an adaptive algorithm selection mechanism. The clustering performance of the K-Means and DBSCAN clustering algorithms is evaluated by calculating silhouette coefficients, and the algorithm with the highest silhouette coefficient is selected to group the original training dataset. The silhouette coefficient is calculated as follows: For any sample x, its silhouette coefficient ; in, Let x be the average distance between sample x and other samples in the same cluster. This is the average distance between sample x and all samples in the nearest neighbor cluster; And / or, In step S1, the text inversion technique involves: freezing the encoder parameters of the text-to-image generation model and optimizing specific learnable labels for each cluster. The initial dimension of the learnable tag is a preset fixed-dimensional floating-point vector. The vector representation of the tag is updated through a preset optimization algorithm so that it can reconstruct the visual semantics of the seed sample to the greatest extent in the text embedding space.

3. The method according to claim 1, characterized in that, In step S2, the adaptive adjustment of the number of test cases generated for each category is calculated using the following distribution-aware formula. Number of classes generated : in, The preset total number of generators, The total number of categories, The set of training sample numbers for each category. For the first The original number of training samples for the class. This represents the maximum number of training samples across all categories. And / or, In step S2, the scene description library contains various real-world environment descriptions and is classified according to scene type or semantic dimension; the construction rule of the text prompt words is: embedding abstract text into learnable tags corresponding to vectors. Combined with description templates from the scene description library, the generated format is "one sheet". The basic prompt for "Photos in [Scene Description]"; for specific categories, add category constraints, and generate a format of " The prompt word is "a photo with a [category name] in the [scene description]"; the scene description library supports dynamic expansion, and the expansion meets the semantic consistency verification rules.

4. The method according to claim 1, characterized in that, In step S3, the feature distance is calculated as follows: the generated test case is input into the pre-trained feature extraction model to obtain the feature vector of the previous layer of the output layer; the Euclidean distance between the feature vector and all cluster centers of the original training data is calculated, and the minimum value is taken as the deviation of the test case; the feature extraction model is any one of ResNet, VGG, AlexNet or MobileNet; And / or, In step S3, the deviation threshold is a dynamic threshold derived from the statistical distribution of training data, which is the 90% or 95% quantile of the feature distance distribution of the training data. The threshold adjustment is triggered by changes in the dataset size or category distribution. When adjusting, the training data distribution is re-statistically analyzed and the threshold is recalculated.

5. The method according to claim 1, characterized in that, When the original training data size is no less than 50,000 images, step S3 uses an approximate deviation calculation method: first, it verifies whether the feature vectors of the training data follow a multivariate normal distribution; if so, it uses... The rule is to calculate the proportion of elements in the generated sample feature vector that exceed 3 times the standard deviation. When this proportion exceeds a preset threshold, it is judged as an invalid test case.

6. An application of the method as described in any one of claims 1-5 in evaluating model defects or performing model retraining, characterized in that, The target deep neural network is input with a valid test set. Samples whose statistical classification output does not match the expected label are used as error samples to evaluate model defects or to retrain the model.

7. A deep neural network fuzzy test case generation system based on cross-modal generation and distribution awareness, characterized in that, include: The abstract modality generation module is used to cluster the original training dataset, select seed samples from each cluster, learn abstract text embedding vectors representing the visual semantic features of the cluster through text inversion technology, and bind them to learnable text tags. The test case generation module is used to construct diverse text prompts based on abstract text embedding vectors combined with a preset scene description library, generate new image test cases using a text-to-image model, and adaptively adjust the number of test cases generated for each category according to the sample quantity distribution of each category in the original training data. The data cleaning module is used to extract the feature vectors of the generated images, calculate the feature distance between them and the cluster centers of the original training data, filter invalid test cases according to the preset deviation threshold, and obtain the valid test set.

8. The system according to claim 7, characterized in that, The abstract modality generation module includes: Clustering units employ either the K-Means clustering algorithm or the DBSCAN clustering algorithm, selecting the optimal clustering algorithm by calculating the silhouette coefficient to group the original training dataset. The seed selection unit selects the most representative seed samples from each cluster according to a preset number or proportion. The selection criteria are quantitative indicators that combine intra-cluster distance and center distance. The text inversion learning unit is used to initialize learnable text tags, freeze the encoder parameters of the text generation image model, and update the vector representation of learnable text tags through optimization algorithms. And / or, The test case generation module includes: The scene library unit is used to store descriptions of real-world environments categorized by scene type or semantic dimension, and supports dynamic expansion and semantic consistency verification. The text generation unit is used to combine learnable text tags with description templates in the scene library to generate diverse text prompts; The distribution-aware control unit is used to statistically analyze the sample quantity distribution of each category in the original training data and calculate the number of test cases generated for each category using the distribution-aware formula. And / or, The data cleaning module includes: The feature extraction unit is used to call a pre-trained feature extraction model to extract feature vectors from the generated image. The similarity calculation unit is used to calculate the Euclidean distance between the generated image feature vector and the cluster centers of the original training data, and to obtain the deviation using an approximate deviation calculation method. The data filtering unit is used to filter invalid test cases based on a dynamic deviation threshold to obtain a valid test set.

9. The system according to claim 7, characterized in that, The abstract modality generation module, test case generation module, data cleaning module, and model evaluation module interact serially or in parallel, and the data transmission format adopts a preset standardized vector format; the system supports distributed deployment, and a load balancing scheduling mechanism is used when multiple GPUs generate test cases in parallel.

10. A deep neural network fuzzy measurement system based on cross-modal generation and distribution sensing, characterized in that, include: The deep neural network fuzzy test case generation system based on cross-modal generation and distribution awareness as described in any one of claims 7-9; The model evaluation module is used to input the valid test set into the target deep neural network, statistically classify the erroneous samples that are inconsistent with the expected labels, and use them to evaluate model defects or perform model retraining.

Images