Multi-modal intelligent fuzzy test input generation method and system based on deep learning

Abstract

The invention discloses a multi-modal intelligent fuzzy test input generation method and system based on deep learning. According to the method, a pre-training model and an abstract syntax tree are adopted to extract multi-modal representation, and a cross-modal encoder is adopted to perform fusion to obtain unified fusion representation; driving a condition generator to generate multi-modal candidate input by taking fusion representation as a condition, implementing directional variation and gradient search around a coverage gain and an abnormal score target in a potential space, and performing format and semantic constraint verification at the same time; and automatically injecting the candidate input into a tested system, identifying an abnormal event, recording and updating, and forming closed-loop optimization. The method is suitable for multi-mode scenes such as texts and images, texts and codes, images and audios, the coverage rate and the new path discovery capability can be remarkably improved on the premise that the performability is guaranteed, false alarms are reduced, and the high-risk vulnerability reaching rate is increased.

Description

Technical Field

[0001] This invention relates to the field of intelligent evaluation technology for digital power grid systems, specifically to a multimodal intelligent fuzzy testing input generation method based on deep learning. Based on domestically developed information technology infrastructure, and oriented towards multimodal inputs such as text and images, text and code, and images and audio, this method effectively improves the efficiency and accuracy of fuzzy testing using deep learning technology. Background Technology

[0002] With the rapid development of information technology, modern software systems are becoming increasingly complex, and more and more applications are beginning to have the ability to process multimodal data. For example, intelligent customer service systems need to simultaneously understand users' voice, text input, and even facial expressions; autonomous driving systems need to integrate information from multiple sensors such as cameras, LiDAR, and GPS; and social media applications are filled with mixed content of text, images, video, and audio. While this multimodal characteristic brings richer functionality and a better user experience to software, it also greatly increases its complexity and potential security risks. Traditional single-modal fuzzing methods typically only test a single type of input, such as plain text or pure binary data, making it difficult to effectively cover and process these emerging multimodal applications. Therefore, researching multimodal intelligent fuzzing input generation methods has significant theoretical and practical value for ensuring the security and robustness of multimodal artificial intelligence systems.

[0003] In recent years, fuzzing techniques combined with artificial intelligence (AI) have made significant progress. Traditional mutation- or generation-based fuzzing methods face challenges in terms of the efficiency and quality of intelligent test case generation. The introduction of AI has brought new possibilities to fuzzing. In test case generation, AI technology can extract features from the target program, train the test case generation model, and thus learn effective test case generation rules. In the field of mobile application GUI fuzzing, the GUIFuzzer framework jointly infers the semantics of GUI widgets by fusing visual features, layout context features, and fine-grained meta-attribute features, and uses deep learning models to optimize GUI event selection strategies, significantly improving fuzzing coverage. The application of Large Language Models (LLMs) in fuzzing is also receiving increasing attention. CKGFuzzer automatically generates fuzzy drivers and optimizes input seeds by combining code knowledge graphs and LLMs. The CHATAFL framework uses LLMs for state transition guidance; when a coverage plateau is detected, it provides the LLM with communication history and requests the generation of messages that can trigger new states. In the field of multimodal fuzzing, besides GUIFuzzer focusing on mobile application GUI testing, research has also applied Multimodal Large Language Models (MLLM) to web application fuzzing. CrawlMLLM utilizes MLLM to assist web crawlers, constructing page state information through parallel analysis of visual and code-based representations, thereby improving the coverage of web application fuzzing. Furthermore, for fuzzing text-to-image (T2I) generation models, researchers have proposed using LLM proxies for automatic generation of jailbreak hints, and the ACTesting method to generate text input by designing mutation operators based on entity-relation (ER) models, combining image object and relation detection to evaluate cross-modal semantic consistency.

[0004] While existing research has made some progress in AI-assisted fuzzing and multimodal application testing, most current work focuses on specific domains or modal combinations, lacking a unified framework to handle diverse multimodal input combinations and generate effective fuzzing inputs. Modern backdoor attacks increasingly tend to distribute triggering logic across multiple computation paths, modalities, or time steps, while existing benchmark suites lack sufficient coverage of multimodal and multipath threats. Therefore, a more general method for generating intelligent multimodal fuzzing inputs is urgently needed to address the increasingly complex security challenges of multimodal applications. Summary of the Invention

[0005] Purpose of the invention: In order to solve the above-mentioned technical problems, the present invention provides a multimodal intelligent fuzzy test input generation method and system based on deep learning, which realizes efficient and reliable generation of fuzzy test input to cope with the increasingly complex security challenges of multimodal applications.

[0006] Technical solution: Firstly, a multimodal intelligent fuzz test input generation method based on deep learning, comprising the following steps:

[0007] S1. For multimodal inputs containing text, images, audio, and code, pre-trained models and abstract syntax trees are used to extract unimodal features respectively. Then, a fusion network based on attention mechanism is used to align and model the unimodal features under different modalities to obtain a unified joint representation.

[0008] S2. The deep generative model is driven by the joint representation to generate candidate inputs, and the mutation operator is used in the latent space to perform targeted mutation to improve the coverage increment and anomaly triggering capability. At the same time, the format and semantic constraint verification is performed, and a high-value seed pool is maintained. The high-value seed is a sample that meets the constraints of the generative model and has a return exceeding a preset threshold.

[0009] S3. Inject candidate inputs into a controlled sandbox environment, collect statement coverage, branch coverage, path coverage, and resource and log operation metrics through instrumentation, construct a composite feedback score to prioritize samples and schedule computational budgets, and write high-value samples that meet specified conditions back to the seed pool to form a closed-loop optimization.

[0010] Furthermore, in step S1, for the image modality, a pre-trained convolutional neural network is used to extract the visual features of the image; for the text modality, a pre-trained language model is used to obtain the context-aware word vectors or sentence vector representations of the text; for the audio modality, an audio feature extraction model is used; and for the code modality, an abstract syntax tree is used to extract the structural and semantic features of the code.

[0011] Let the unimodal representations of text, image, audio, and code be as follows: Where m represents the length of the text modality input sequence, n represents the number of visual feature vectors obtained after the image modality is processed by the convolutional neural network, p represents the number of time frames or feature segments obtained after feature extraction of the audio modality, and q represents the number of syntax tree nodes or code structural units of the code modality. , , and These represent the original dimensions of text, image, audio, and code modal features, respectively.

[0012] We obtain this through linear mapping: , ,in, , , and These represent linear mapping matrices for text, image, audio, and code modalities, respectively, used to map vectors from their original dimensions to a unified dimension d;

[0013] Building features , Encode position and modality type, and... Input to a cross-modal encoder to obtain the output The joint characterization is obtained through convergence. .

[0014] Furthermore, in step S2, the deep generative model includes a conditional generator. With discriminator The generator learns to output samples that are reasonable but may reach boundary behaviors across various modalities. The discriminator distinguishes between real and generated samples and can be extended to predict anomalous tendencies. Through adversarial or variational training, a diverse and effective candidate set is formed, satisfying the following:

[0015] ;

[0016] ;

[0017] in, To obtain a joint representation based on multimodal inputs, It is random noise. For seed library embedding vectors, For the generated candidate input, For discriminator Based on real samples and The output multi-metric vector includes fidelity. Incremental Coverage With anomaly probability measure.

[0018] Furthermore, generator The specific implementation is one or a combination of a generator of a conditional generative adversarial network, a decoder of a conditional variational autoencoder, or a self-attention sequence generator with the decoder as the main component.

[0019] when When using a generator with a conditional generative adversarial network, the representation J, random noise Z, and seed library embedding vector are fused. After being concatenated or linearly mapped, the data is input into the generator network, where candidate samples for each modality are generated through deconvolution, feedforward networks, or attention modules. ;

[0020] when When using a conditional variational autoencoder in the decoder, the encoder first bases the decoder on the fused representation J and the seed library embedding vector. The latent variable distribution parameters are obtained, and then the latent variable vector is generated using reparameterization techniques. The decoder then performs controlled sampling and continuous interpolation in the latent space to generate cross-modal candidate samples with a coherent structure. ;

[0021] when When using a decoder-centric self-attention sequence generator, the fused representation J is used as the conditional context, along with a random perturbation vector or a seed embedding vector. Cross-modal sequences or structured data are generated through a multi-head self-attention mechanism, and candidate samples are obtained through stepwise sampling or autoregressive generation. ;

[0022] When using a combination approach, The three types of generation architectures generate their respective candidate sample fragments, and the results of the three are aggregated in the output stage to form a unified multimodal candidate sample set.

[0023] Furthermore, in step S2, the operators for directional mutation include one or more of the following: gradient ascent search on the potential vector or output representation, parameter perturbation, masking and time shifting, amplitude perturbation, structural transformation based on abstract syntax tree, and equivalent interface replacement. The operators are selected probabilistically weighted based on historical returns to balance exploration intensity and effectiveness.

[0024] Establish a set of mutation operators for each mode. The operator sampling probability is given by the following formula:

[0025]

[0026] Among them, historical returns By operator The composite feedback signals obtained in the previous several rounds of generation-execution-feedback loops are accumulated or weighted averaged to obtain the result. The threshold value is used.

[0027] Furthermore, in step S2, the format and semantic constraint verification includes verifying the legality of the input structure, the range of values, and the temporal dependencies, and triggering lightweight repair when the constraints are not met so that the candidate input can be parsed and executed by the system under test.

[0028] Further, in step S3, the composite feedback score is calculated using the following formula:

[0029] ;

[0030] in, Indicates the sample Composite feedback score, Indicates execution sample The weighted coverage increment, , For collaborative coverage weighted matrix, For the sample A single execution yields a coverage metric: , To collect collaborative coverage metrics on the multimodal model side: , For augmenting the coverage vector: , To cover the baseline, These represent the number of statement-level, branch-level, and path-level coverages reached after executing sample x. The branch-level coverage reflects the hit rate of program conditional branches, while the path-level coverage measures the exploration depth and diversity of program control flow paths. These represent the fusion layer hit rate, attention head activity rate, and key operator activation rate, respectively. For resource vectors, As an indicator of novelty or rarity, The consistency conflict intensity is derived from the breach metric of cross-modal matching and format validation; This represents the probability of an anomaly. For weight parameters; based on Perform priority and budget scheduling and update the seed pool.

[0031] Furthermore, based on When performing priority and budget scheduling, the scheduling probability is: Where T is the temperature parameter to balance exploration and utilization;

[0032] When the sample satisfies the constraint verification function and When the score exceeds a preset threshold, the seed pool is updated. ; and will and This signal is sent back to the upstream strategy as a reward. This indicates that a deduplication operation is performed on the candidate sample set.

[0033] Secondly, a deep learning-based multimodal intelligent fuzz test input generation system includes:

[0034] The feature extraction and fusion submodule is used to extract single-modal features from multimodal inputs containing text, images, audio and code using pre-trained models and abstract syntax trees respectively, and to align and model the single-modal features under different modalities through a fusion network based on an attention mechanism to obtain a unified joint representation.

[0035] The multimodal input generation and mutation submodule is used to drive the deep generative model to generate candidate inputs based on the joint representation as a condition, and to implement targeted mutation using mutation operators in the latent space to improve coverage increment and anomaly triggering capability. At the same time, it performs format and semantic constraint verification and maintains a high-value seed pool, wherein the high-value seeds are samples that meet the constraints of the generative model and have a return exceeding a preset threshold.

[0036] The fuzz test execution and feedback submodule is used to inject candidate inputs into a controlled sandbox environment. It collects statement coverage, branch coverage, path coverage, and resource and log runtime metrics through instrumentation, constructs a composite feedback score for sample priority and computational budget scheduling, and writes high-value samples that meet specified conditions back to the seed pool to form a closed-loop optimization.

[0037] Furthermore, the feature extraction and fusion submodule specifically includes: a modality-aware feature extraction unit, used to extract features from text, images, audio, and code respectively; for text, a pre-trained language model is used to obtain contextual semantic representations; for images, a convolutional network or visual Transformer is used to extract multi-scale visual representations; for audio, time-frequency domain features are extracted; and for code, structural and semantic representations are obtained based on abstract syntax trees and code pre-trained models; an infrastructure adaptation unit, used to adapt interfaces and replace operators for processors, operating systems, and compilation and inference frameworks to ensure the usability and performance of the feature extraction process in real-world environments; a cross-modal alignment and fusion encoding unit, used to employ early fusion, late fusion, and hybrid fusion strategies combined with multi-head attention and cross-attention to complete feature association modeling and output a unified fusion representation; and a consistency constraint and knowledge base unit, used to introduce constraints such as cross-modal matching, reconstruction, and contrastive learning, and maintain a rule base such as format specifications, value ranges, and temporal dependencies to verify the fusion representation and improve discriminability and robustness.

[0038] Furthermore, the deep learning-based multimodal input generation and mutation submodule specifically includes: a conditional generation unit, used to generate candidate test inputs by driving the generative model with fused representations as conditions. The generative model includes generative adversarial networks, variational autoencoders, and sequence-to-sequence models based on Transformers, and incorporates semantic and format validity checks to ensure that the tested system is parseable; a latent space directional search optimization unit, used to perform gradient, evolutionary, and heuristic searches in the latent space of the generative model, with the joint objectives of coverage increment, anomaly score, and sample diversity for directional exploration; a mutation operator library adaptive selection unit, used to provide multimodal mutation operators such as parameter perturbation, local occlusion, time or frequency dimension shift, addition, deletion, and modification of cue fragments, syntax tree level code transformation, and equivalent interface replacement, and to perform probability-weighted selection of operators based on historical returns to balance exploration and utilization; and a high-value seed maintenance unit, used to perform difference minimization, deduplication, and labeling of samples that can trigger new paths or anomalies into a library, forming a reusable high-quality seed set and providing prior reference for subsequent generation strategies.

[0039] Furthermore, the fuzzing execution and feedback submodule specifically includes: a sandbox execution and resource isolation unit, used to batch inject and execute candidate inputs in a controlled environment, providing process, network, and file-level isolation as well as resource quotas and timeout control, and adapting to the security policies and interfaces of domestic operating systems; an instrumentation and coverage collection unit, used to collect code coverage information at the statement, branch, and path levels, and collect collaborative coverage metrics such as fusion layer, attention head, and key operator activation on the multimodal model side; a runtime monitoring unit, used to capture crashes, assertion failures, timeouts, abnormal returns, logs, and resource usage trajectories in real time, generate reproducible experimental records, and complete audit evidence such as timestamps and signatures; and a composite feedback scoring unit, used to construct a comprehensive score based on coverage increment, anomaly rarity, consistency conflict strength, and execution cost, dynamically adjust sample priority and computational budget, and feed it back to the seed library and generation strategy to achieve continuous optimization and closed-loop scheduling.

[0040] Thirdly, the present invention also provides an electronic device, comprising: a processor; a memory; and a computer program stored in the memory and configured to be executed by the processor, wherein the computer program, when executed by the processor, implements the steps of the deep learning-based multimodal intelligent fuzzy test input generation method as described in the first aspect of the present invention.

[0041] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the deep learning-based multimodal intelligent fuzzy test input generation method as described in the first aspect of the present invention.

[0042] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep learning-based multimodal intelligent fuzzy test input generation method as described in the first aspect of the present invention.

[0043] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] (1) By combining multimodal feature fusion with conditional generation and implementing directional mutation and target constraints in the latent space, the test input is guided to evolve in a direction that is more sensitive to coverage increment and abnormal triggering. This can effectively break through the blind spots that are easy to occur in a single modality and improve the ability to reach hidden defects across modalities and multiple paths.

[0045] (2) Establish a composite feedback score based on coverage increment, rareness of anomalies and consistency conflict intensity, and combine energy scheduling and seed priority management to significantly reduce invalid exploration and duplicate calculations, and achieve higher sample output efficiency and faster convergence speed under limited computing power.

[0046] (3) Set up semantic and format protection verification and lightweight repair mechanism to ensure that the generated and mutated inputs meet the parsing and constraint requirements, reduce the proportion of filtered samples in the early parsing stage, reduce false alarms and scrapped samples, and steadily improve the executability and effective triggering rate.

[0047] (4) Through the closed-loop collaboration of feature extraction and fusion, condition generation and targeted mutation, and execution and feedback, the coverage increment, the number of unique vulnerabilities discovered and the ability to reach high-risk defects were empirically improved, and good transferability and engineering robustness were demonstrated in multi-scenario tasks. Attached Figure Description

[0048] Figure 1 This is a flowchart of the deep learning-based multimodal intelligent fuzzy test input generation method in this invention;

[0049] Figure 2 This is an architecture diagram of the deep learning-based multimodal intelligent fuzzy test input generation system in this invention;

[0050] Figure 3 This is a diagram illustrating the overall design framework of the deep learning-based multimodal intelligent fuzzy test input generation method in this invention.

[0051] Figure 4 This is a flowchart of the multimodal input feature extraction and fusion module;

[0052] Figure 5 This is a flowchart of a deep learning-based multimodal input generation and mutation module;

[0053] Figure 6This is the flowchart for the fuzz test execution and feedback module. Detailed Implementation

[0054] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0055] In this embodiment of the invention, the deployment is oriented towards domestic infrastructure. The environment includes domestic processors, domestic operating systems, and domestic compilation and inference frameworks. Through a closed-loop framework consisting of three parts—multimodal feature extraction and fusion, input generation and mutation, and execution and feedback—efficient generation and optimization of multimodal intelligent fuzzy test inputs are achieved.

[0056] Feature extraction and fusion stage: Extract effective features from input data of different modalities and learn the intrinsic relationships between them. The goal is to transform the original multimodal input data into a unified representation rich in semantic information, laying the foundation for subsequent intelligent input generation and variation.

[0057] For each input modality, a suitable deep learning model needs to be designed or selected for feature extraction. For image modality, a pre-trained convolutional neural network is used to extract visual features of the image; for text modality, a pre-trained language model is used to obtain context-aware word vectors or sentence vector representations of the text; for audio modality, an audio feature extraction model is used. For code modality, an abstract syntax tree (AST) is used to extract structural and semantic features of the code.

[0058] After extracting features from each modality, the next crucial step is feature fusion. The goal of feature fusion is to integrate these features from different sources, with varying dimensions and distributions, into a consistent and effective joint representation. The choice of fusion strategy is critical; common methods include early fusion, late fusion, and hybrid fusion. In this approach, considering the need to provide rich cross-modal information for the subsequent generative model, a Transformer encoder is used to learn the interactions and dependencies between features from different modalities. Specifically, the feature vectors extracted from each modality are concatenated, added, or mapped through a fully connected layer before being input into a fusion network. This network can learn the contribution weights of different modal features to the final task and dynamically adjust the fusion method. When processing mixed image and text inputs, the model can learn which regions in the image are associated with which words in the text description, thus generating a more consistent fused representation. The fused representation should capture the overall semantics of the multimodal input and remain sensitive to subtle changes in each modality, which is crucial for generating effective fuzzy test inputs. In addition, to improve the fusion effect, auxiliary tasks such as contrastive learning or cross-modal reconstruction are introduced to enhance the discriminativeness and robustness of the fused representation.

[0059] Input generation and mutation phase: Utilize deep learning models to generate new and diverse test cases based on learned multimodal representations, and support intelligent mutation of existing seeds.

[0060] Deep learning-based multimodal input generation and mutation is the core component of this framework responsible for generating diverse test inputs with potential vulnerability triggering capabilities. This stage receives joint representations from the feature extraction and fusion stages and utilizes deep generative models to create new multimodal input samples. Considering the complexity and diversity of multimodal data, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer-based sequence-to-sequence models are preferred generative architectures. A Conditional Generative Adversarial Network (cGAN) is designed, where the generator, conditioned on the fused representations, learns to generate test inputs that appear reasonable across modalities but may contain hidden anomalies. The discriminator is responsible for distinguishing generated inputs from real inputs and can be further extended to predict whether the input will cause program anomalies. Through adversarial training between the generator and discriminator, the model can gradually learn to generate multimodal inputs that are increasingly difficult for the target system to process. Another possible approach is to use a VAE, which learns the latent distribution of the input data, samples from it, and decodes it to generate new samples. The advantage of VAEs lies in the fact that their latent space usually has good mathematical properties, making it easy to interpolate and perturb, thereby generating smoothly transitioning variant inputs.

[0061] The introduction of mutation strategies is crucial in the generation process. Simple generation may not be effective in exploring the boundary conditions of the program. Therefore, targeted mutations need to be introduced based on generation. One approach is to explore and perturb the latent space of the generative model. Small random noise can be added to the latent vectors, and the search can proceed along directions in the latent space that maximize objective functions, such as code coverage gain or anomaly scores. These mutation operations can be combined with deep learning generative models; for example, a model can be trained to predict which types of mutations are more likely to cause program errors, or to learn how to apply mutation operations to generated samples to enhance their destructiveness. Furthermore, the idea of ​​adversarial attacks can be borrowed, using the gradient of the input loss relative to the target model to guide small perturbations of the input, enabling them to deceive the model or trigger unexpected behavior. For example, an objective function can be designed to maximize when the program covers a new path or triggers a crash, and then the input can be iteratively modified using gradient ascent. The generated test inputs need to meet the format constraints and basic semantic requirements of each modality to avoid being filtered out in the early parsing stages of the target system, thus enabling in-depth testing of the core logic.

[0062] Execution and Feedback Phase: The feedback mechanism dynamically adjusts the generation strategy based on test results, such as code coverage and crash information, guiding the fuzzing process to explore code areas that are more likely to contain vulnerabilities.

[0063] The fuzzing execution and feedback mechanism is a crucial closed-loop system ensuring the continuous optimization of the entire framework and effective vulnerability discovery. This mechanism is responsible for actually submitting the multimodal test cases generated by the generation and mutation modules to the system under test (SUT) and closely monitoring the system's behavior when receiving and processing these inputs. The execution process needs to be automated and capable of handling various types of multimodal inputs; for example, combining text and images into an image annotation system, or inputting code and text descriptions into a code generation model. During execution, effective monitoring methods need to be deployed to capture various runtime information of the system. This information includes, but is not limited to: code coverage (e.g., branch coverage, statement coverage, path coverage, etc., which can be obtained through instrumentation techniques), system resource usage (e.g., CPU, memory usage), program output, log information, and whether crashes, assertion failures, timeouts, or other unexpected behaviors have occurred. This monitoring data constitutes the main source of feedback information.

[0064] The most direct use of collected feedback information is vulnerability detection. Once a system crash or other serious anomaly is detected, relevant test inputs and program state information are recorded as evidence of potential vulnerabilities for subsequent vulnerability analysis and confirmation. Secondly, feedback information guides the prioritization of test cases and seed selection. For example, test cases that trigger new code paths, improve code coverage, or cause rare system behaviors are generally considered more valuable, should be given higher priority, and may be added to a high-quality seed pool for subsequent mutation and generation to discover more related vulnerabilities. Thirdly, feedback information can be used to optimize generation and mutation modules. For example, if a certain type of mutation strategy or generation pattern is found to generate more effective test cases, the parameters of the generation model or the weights of mutation operations can be adjusted to favor these more effective directions. This can be seen as an online learning process, where the generation model continuously adjusts its behavior based on feedback from test results. Fourthly, feedback information can also be used to evaluate the overall effectiveness of fuzz testing. For example, by tracking metrics such as the growth curve of code coverage, the number of vulnerabilities discovered, and their severity, the effectiveness of the current testing strategy can be evaluated, and adjustments can be made as necessary. An efficient feedback mechanism can ensure that the fuzzing process is not blind, but can be dynamically adjusted and continuously improved based on the actual test results, thereby maximizing the efficiency and depth of vulnerability discovery.

[0065] Reference Figure 1 The deep learning-based multimodal intelligent fuzzy test input generation method comprises steps S1, S2, and S3 from top to bottom, with iterative optimization achieved through a feedback loop. In the feature extraction and fusion stage (S1), step S11 is executed first to extract representations from each modality: for images, a pre-trained convolutional neural network is used to acquire visual features; for text, a pre-trained language model is used to obtain contextual semantic representations; for audio, an audio feature extraction model is used to generate time-frequency domain representations; and for code, structural and semantic information is extracted based on an abstract syntax tree. Step S12 is then executed, mapping the features of each modality to a unified space and inputting them into the fusion network. A Transformer encoder is used to learn cross-modal interactions and dependencies, enabling joint modeling through early fusion, late fusion, or hybrid fusion. Simultaneously, auxiliary tasks such as contrastive learning or cross-modal reconstruction are introduced to enhance the discriminativeness and robustness of the fused representation. Finally, a unified and semantically rich information representation is obtained, serving as the conditional input for subsequent generation and mutation.

[0066] In the input generation and mutation phase S2, S21 is first executed, where multimodal candidate inputs are generated using a deep generative model architecture, conditioned on fused representations. This deep generative model architecture includes conditional generative adversarial networks, variational autoencoders, and Transformer-based sequence-to-sequence models. These are optional generative model families, and one or a combination of them can be selected based on the target scenario. The generator of the generative model learns to output candidate samples that have structural rationality across modalities but can reach boundary behaviors. The discriminator distinguishes between real and generated samples and can be extended to predict anomalous tendencies, forming a diverse and effective candidate set through adversarial training or variational training mechanisms. Subsequently, S22 is executed, performing targeted mutations based on the generated samples: small perturbations are added to the latent space, and a directional search is performed along objectives such as coverage gain and anomalous scores, which can be combined with gradient methods for iterative modification. Simultaneously, semantic and format constraints are set to ensure that the samples can be parsed and executed by the target system, thereby improving the ability to explore program boundary conditions and trigger anomalies.

[0067] In the S3 execution and feedback phase, S31 is executed first, automatically injecting candidate inputs into the system under test and collecting metrics such as statement coverage, branch coverage, and path coverage through instrumentation and other methods. Simultaneously, resource usage, program output, logs, and runtime events such as crashes, assertion failures, and timeouts are monitored to form complete test evidence. Next, S32 is executed, conducting vulnerability detection and evidence recording based on the feedback obtained. Test cases that can trigger new paths or anomalies are prioritized and added to a high-quality seed library to guide subsequent generation and mutation. Simultaneously, the generation model parameters and mutation weights are adjusted online based on the effectiveness of different strategies as part of an adaptive learning process. This feedback is also used to evaluate the overall testing effect and drive continuous process iteration until coverage and anomaly diversity thresholds or resource budget limits are reached.

[0068] exist Figure 1 Based on the aforementioned method and flow, to facilitate engineering deployment in domestic processor and operating system environments, the system-level working mechanism is explained below. See [link to documentation]. Figure 2 The system structure diagram of the multimodal intelligent fuzzy test input generation system based on deep learning is shown. The system is functionally divided according to the three stages in the method. Each module corresponds to step S1 to S3. The system forms an end-to-end closed loop by fusing representations, candidate inputs and feedback metrics between modules.

[0069] The feature extraction and fusion submodule is used to accept multi-source inputs and construct a unified semantic representation. It calls pre-trained models to extract effective features from text, image, audio, and code modalities, and adapts interfaces and operators to domestic processors and operating systems to ensure usability and performance in domestic IT environments. After the features from each modality are mapped to a unified space, attention-based fusion encoding is used to model cross-modal dependencies. Consistency and reconstruction constraints are combined to enhance the discriminativeness and robustness of the representation, ultimately outputting a fused representation that can be directly consumed by subsequent modules.

[0070] The multimodal input generation and mutation submodule generates candidate inputs for the test target based on fused representations and expands the exploration depth. This module uses fused representations as conditions to drive the generative model for sample synthesis. The model can be a conditional generative adversarial network, variational autoencoder, or a self-attention-based sequence model to obtain semantically reasonable initial samples across modalities. Based on the generation, targeted search and mutation of the latent space are introduced, performing gradient or heuristic optimization around objectives such as coverage gain and anomaly scores, and performing validity verification through format and semantic safeguards. Samples that can trigger new paths or anomalies are minimized and accumulated as high-value seeds, providing reusable priors for subsequent iterations.

[0071] The fuzzing execution and feedback submodule injects candidate inputs into the system under test and transforms operational evidence into usable feedback. This module automates sample execution within a controlled sandbox, collecting metrics such as statement coverage, branch coverage, and path coverage, and recording resource usage, output logs, and events such as crashes, assertion failures, and timeouts. Based on these metrics, a composite feedback score is constructed to drive the dynamic allocation of sample priorities and computational budgets. Effective information is then fed back into the generation and mutation modules to adjust the objective function and operator weights, while simultaneously updating the high-value seed library, achieving a closed loop of continuous optimization.

[0072] With the system architecture clearly defined, the overall design framework is explained below to facilitate understanding of how this method operates in engineering. (See also...) Figure 3 The overall design framework diagram of a deep learning-based multimodal intelligent fuzzy testing input generation method is shown. The raw data from the multimodal input samples first enters the text feature extraction, image feature extraction, audio feature extraction, and code feature extraction processes along four parallel branches. The representations extracted from each branch are then fed into a feature extraction and fusion module to complete cross-modal alignment and joint encoding, outputting a unified fused representation. This fused representation serves as the conditional input for subsequent generation and mutation, and is the data baseline connecting the perception and generation sides.

[0073] On the generation side, the fused representations are fed into the multimodal input generation and mutation submodule. Under the adversarial learning training paradigm, this module synthesizes candidate samples and performs structural perturbations based on three optional deep generative model architectures. These three optional model architectures include Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer-based sequence-to-sequence models. The system can choose to use one of these individually or in combination, depending on the characteristics and modality of the test task. The GAN generator learns a deceptive sample distribution in the multimodal space, the VAE decoder performs controlled sampling and continuous interpolation in the latent space, and the Transformer handles cross-modal mapping and conditional generation in sequence-to-sequence form. When a combination approach is used, candidate samples are generated by GANs, VAEs, and Transformer-based sequence-to-sequence models respectively, and their outputs are then converged to form a unified candidate set.

[0074] The module outputs initial candidate samples under the constraints of fused representations and performs directed mutations in the latent space to expand the exploration scope, resulting in a diverse set of samples oriented towards the test target.

[0075] On the validation and closed-loop side, candidate samples are fed into the fuzz test execution and feedback module, where they are injected and interact with the target system to form metric evidence. Key metrics returned by the target system include code coverage, anomaly detection results, and crash information. These metrics are aggregated into the feedback optimization path and, after comprehensive evaluation, used to update the search direction and model hyperparameters of the generation and mutation modules. Simultaneously, the parameter selection of the feature extraction and fusion modules is fine-tuned, and the training plan and sample allocation for adversarial learning are adjusted as needed. Through directed data flow between multimodal input samples, the feature extraction and fusion module, the deep learning generation and mutation module, the fuzz test execution and feedback module, the target system, and the feedback optimization, Figure 3 The framework shown completes a closed-loop operation from representation construction to sample synthesis, from operational verification to policy updates, ensuring that the system can continuously improve its coverage increment and anomaly detection capabilities in the context of information technology innovation.

[0076] exist Figure 1 The method and process are as follows Figure 2 Based on the system architecture, to explain the internal working mechanism of the feature extraction and fusion stages, we will now combine... Figure 4 The workflow of the multimodal input feature extraction and fusion submodule is described. This module accepts four types of input: text, image, audio, and code. It sets up four independent but collaborative units, each responsible for the representation extraction task of its respective modality. Cross-modal association modeling is completed in the fusion network within the module, thereby outputting a unified and measurable joint representation to the subsequent generation and mutation stages.

[0077] Module 1: Feature Extraction and Fusion Submodule

[0078] This submodule includes a modality-aware feature extraction unit, a domestic infrastructure adaptation unit, a cross-modal alignment and fusion coding unit, and a consistency constraint and knowledge base unit, which collaboratively complete the construction from the original multimodal input to a unified joint representation. The modality-aware feature extraction unit performs representation learning for text, images, audio, and code respectively. Let the single-modal coding results for text, images, audio, and code be as follows:

[0079]

[0080] To facilitate subsequent fusion, each modality is aligned to the same dimension d through a linear mapping:

[0081]

[0082] Domestic Infrastructure Adaptation Unit: Equivalent replacement mapping for execution operators and interfaces for domestic processors, domestic operating systems, and domestic compilation and inference frameworks. And through kernel parameters and scheduling policy set To ensure availability and performance stability in a domestically developed information technology environment, i.e., for any operator ,have

[0083]

[0084] Cross-modal alignment and fusion coding units will Concatenate the data and add the position and modality type encoding P, then input it into the cross-modal Transformer fusion network:

[0085]

[0086] The focus of the bullish attention is:

[0087]

[0088]

[0089] In cross-modal Transformer fusion networks, the query vector of the multi-head attention mechanism Key vector AND value vector All are composed of input sequences Obtained through their respective linear projections. Specifically, After three sets of learnable projection matrices They were then obtained respectively Therefore, all modalities participate in attention computation in a unified representation space, and the Q / K / V of each attention head are generated based on the fused cross-modal sequence, thereby realizing global interaction between cross-modalities.

[0090] Depending on the task requirements, early fusion corresponds to the direct cascading of X0, while late fusion corresponds to weighted pooling after each modality. Create a weighted portfolio:

[0091]

[0092] Hybridization will With Transformer output X L The global pooling results are concatenated and then projected to obtain a unified joint representation:

[0093]

[0094] Consistency Constraints and Knowledge Base Unit: Maintaining a Set of Formatting Specifications Set of Value Ranges With time-dependent sets And verify the function through constraints:

[0095]

[0096] The fusion results are validated, and contrastive learning and cross-modal reconstruction are introduced as auxiliary tasks to improve discriminativity and robustness.

[0097]

[0098]

[0099]

[0100] in, A cross-sample contrastive learning loss is used to improve the discriminativeness of the fused representation in the multimodal space; The reconstruction loss measures the deviation between the fused representation reconstructed from the original input by each modal decoder. To contrast the temperature coefficient in learning, the sensitivity of contrast loss is controlled by adjusting the smoothness of the similarity distribution.

[0101] symbol Indicates modality The decoding mapping function will fuse the representation. Restore or reconstruct the input space to the corresponding modality; and These represent the fused representations used to construct positive and negative sample pairs in contrastive learning, where... and Similarity between It is calculated using cosine similarity or dot product similarity and is used to measure the distance relationship between samples in a unified feature space.

[0102] Final output As a conditional input for subsequent generation and mutation modules, it retains key semantics while remaining sensitive to subtle perturbations, thus providing a stable and reliable representation basis for targeted exploration and feedback optimization.

[0103] Module 2: Deep Learning-Based Multimodal Input Generation and Mutation Module

[0104] exist Figure 4 The obtained unified joint representation Based on this, the generation and mutation module of this embodiment is as follows: Figure 5 The module operates as shown. It comprises four units: a condition generation unit, a latent space directional search optimization unit, a mutation operator library adaptive selection unit, and a high-value seed maintenance unit. The module inputs include J from the fusion module and optional random noise. And a seed pool consisting of historically high-value examples. .

[0105] Condition generator in the condition generation unit by Conditional generation of multimodal candidates Formalized as:

[0106]

[0107] in This is an embedding aggregation of a seed subset. The specific implementation can be a generator of a conditional generative adversarial network, a decoder of a conditional variational autoencoder, or a self-attention sequence generator with the decoder as the main component.

[0108] when When using a conditional generative adversarial network (cGAN) generator, the representation J, random noise Z, and seed embedding vector are fused. After being concatenated or linearly mapped, the data is input into the generator network, where candidate samples for each modality are generated through deconvolution, feedforward networks, or attention modules. This implementation method is suitable for scenarios that require emphasizing sample deception, structural diversity, or adversarial behavior.

[0109] when When using a conditional variational autoencoder (cVAE) decoder, the encoder is first based on J and The latent variable distribution parameters are obtained, and then the latent variable vector is generated using reparameterization techniques. The decoder then performs controlled sampling and continuous interpolation in the latent space to generate cross-modal candidate samples with a coherent structure. This approach is suitable for scenarios where the interpretability or adjustability of the latent space is highly required.

[0110] when When using a decoder-centric Transformer sequence generator, the fused representation J is used as the conditional context, along with random perturbation vectors or seed embeddings. A multi-head self-attention mechanism is employed to generate cross-modal sequences or structured data, and candidate samples can be obtained through stepwise sampling or autoregressive generation. .

[0111] When using a combination approach, Candidate sample fragments can be generated by three generative architectures: cGAN, cVAE, and Transformer. The results from these three architectures are then aggregated at the output stage to form a unified multimodal candidate sample set. This parallel generation and aggregation method simultaneously preserves the structural perturbation capability of cGAN, the latent space continuity of cVAE, and the sequence modeling capability of Transformer, thereby improving the overall generation quality and sample space coverage.

[0112] Built-in semantic and format validity validation functions right Perform rapid screening. Only then can we proceed to the next stage. In adversarial learning scenarios, the discriminator... Receive real samples and Output multi-index vector:

[0113]

[0114] These represent fidelity, coverage gain, and anomaly probability, respectively, and are used for training signals and subsequent scheduling.

[0115] The latent space directed search optimization unit, based on generation, employs a mutation strategy guided by deep learning, consisting of a mutant. This involves targeted perturbations of the latent variables or output representation to expand the exploration boundary. Let... Represents the candidate samples generated from the latent vector z, with the combined objective function as:

[0116]

[0117] in , To measure sample diversity, a projected gradient ascent search is employed.

[0118]

[0119] in For projection operators to satisfy format and semantic constraints, The step size.

[0120] The mutation operator library adaptively selects units to establish a set of mutation operators for each mode. This covers parameter perturbation, partial occlusion, time or frequency offset, addition, deletion, and modification of prompt fragments, syntax tree level code transformation, and equivalent interface replacement. Based on historical returns... Calculate operator sampling probability and historical returns. By operator This feedback signal is obtained by accumulating or weighting the composite feedback signals obtained in the previous generation-execution-feedback loops. The feedback signal comes from the coverage increment of the executed samples. Anomaly probability Novelty index and resource costs Multiple metrics are used to measure the overall contribution of operators in exploring new paths, triggering anomalies, or improving sample diversity.

[0121] The operator sampling probability is given by the following formula:

[0122]

[0123] and according to Selecting operators and applying mutations to candidates yields... Language model-based rewrite operators can be superimposed on text and code. It includes structure-preserving operators such as image and audio overlay masking, time shifting, and amplitude perturbation.

[0124] High-value seed maintenance units are trained and screened using a combination of adversarial loss and compound reward. On one hand, a discriminant and feedback mechanism is used to drive parameter updates and sample scheduling through multi-objective learning and adversarial objectives, employing adversarial loss:

[0125]

[0126] in, This indicates the distribution of data from the real data. samples Perform expectation calculations to measure the output statistics of the discriminator on real samples; Indicates random noise and fusion characterization The expectation operation is performed on the joint distribution to evaluate the generator under conditional input. The statistical behavior of the generated samples under the influence of adversarial loss. These two correspond to the expected values ​​of the real and generated sample terms in the adversarial loss, respectively.

[0127] On the other hand, a composite reward consisting of discriminator metrics is introduced:

[0128]

[0129] This serves as the basis for sample prioritization and budget allocation, and minimizes the loss due to cross-modal consistency and format validation constraints. :

[0130]

[0131] Within the adversarial learning framework, the generative model and discriminator perform minimization and maximization optimizations, respectively: the generative model minimizes the joint objective consisting of adversarial loss, a composite reward term, and a consistency constraint term to improve the effectiveness and usability of generated samples; the discriminator maximizes the adversarial loss to enhance its ability to distinguish between real and generated samples. These two aspects work alternately to form a game-like training process, thereby driving the system to stably generate high-value candidate samples.

[0132] when and At that time, perform difference minimization and deduplication, and then insert the data into the database:

[0133]

[0134] Simultaneously update operator returns The model parameters are used to support subsequent iterations.

[0135] Ultimately, the constraints are satisfied and a high level of performance is achieved. High-value samples are written back to the seed pool. The remaining samples are used for further iterations or discarded; each modality outputs... After decoding and legality checks, the data is processed into text, images, audio, and code candidates, which are then used in the subsequent execution and feedback stages.

[0136] Module 3: Fuzz Test Execution and Feedback Module

[0137] After outputting candidate samples, the execution and feedback submodule follows... Figure 6 The sequence shown forms a closed loop around sandbox execution and resource isolation, instrumentation and coverage acquisition, runtime monitoring, and composite feedback scoring. Multimodal samples from the test case pool are denoted as:

[0138]

[0139] An automatic execution engine adapted to the security policies and interfaces of domestic operating systems. The process is injected into a controlled environment to complete execution; this environment employs containerization or hardware emulation and provides process, network, and file-level isolation, as well as resource quotas and timeout control, denoted by the resource upper limit vector. Parallel access at compile time or binary level instrumentation. .

[0140] During one execution, the sample A single execution yields a coverage metric. ,

[0141] Simultaneously, collaborative coverage metrics are collected on the multimodal model side. (Representing the fusion layer hit rate, attention head activation rate, and key operator activation rate, respectively). Let the augmented coverage vector... Its baseline The weighted coverage increment is defined as:

[0142]

[0143] in This is a weighted matrix for collaborative coverage.

[0144] The runtime monitoring unit records performance vectors.

[0145]

[0146] And determine resources and timeout events. ; Anomaly Indication Events such as crashes and assertion failures are identified. Execution logs, I / O data, and stack backs are organized into structured records. And attach a timestamp and signature to complete the audit evidence: .

[0147] The composite feedback scoring unit calculates a comprehensive score for each sample based on the above metrics and drives scheduling and backfeeding. Let the consistency conflict strength be... (Based on the breach metric derived from cross-modal matching and format validation), then there is a single sample Overall rating:

[0148]

[0149] in For anomaly probability estimation, As an indicator of novelty or rarity, and These are learnable weights.

[0150] in accordance with By allocating priorities and budgets, the scheduling probability can be written as:

[0151]

[0152] Where T is the temperature parameter to balance exploration and utilization.

[0153] When the sample satisfies the constraint verification function and At that time, the seed pool is updated.

[0154]

[0155] And and This signal is passed back to the upstream strategy as a reward.

[0156] Simultaneously, the module provides numerical rewards to the generator and mutant to support online adaptive fine-tuning of the generator and mutant parameters:

[0157]

[0158] With generator parameters For example, using a reward-weighted update:

[0159]

[0160] If necessary, increase the learning rate or enable continuous learning strategies to accelerate adaptation; simultaneously, anomaly and coverage evidence are processed through... These are solidified into traceable entries, enabling closed-loop refeeding of high-value samples. This is achieved through a continuous pathway: "Use Case Pool → Automated Execution Engine and Instrumentation → Runtime Monitoring → Result Aggregation → Feedback Decision → Rewards and Updates." Figure 6 The module shown transforms operational evidence into optimizable signals and continuously drives the evolution of the seed bank and the improvement of model strategies while maintaining format and semantic constraints.

[0161] To verify the effectiveness of the proposed deep learning-based multimodal intelligent fuzzing input generation method, a unified experimental environment was constructed, and comparative and ablation evaluations were conducted. Experiments were performed on a server configured with a high-performance NVIDIA A100 processor to support the training and inference of deep learning models. The software environment included mainstream deep learning frameworks such as TensorFlow and the fuzzing toolchain AFL++ as baselines or components, along with relevant dependency libraries. Dataset selection was tailored to different target multimodal combinations. For example, for text and image combinations, publicly available datasets containing images and their corresponding text descriptions, such as COCO Captions and Flickr 30k, were used; for text and code combinations, the code library Code Search Net, containing code snippets and their natural language descriptions, and datasets collected from open-source projects such as GitHub were used; for image and audio combinations, datasets containing video (which can be decomposed into image frames and audio streams) and their descriptions, such as AudioSet and ActivityNet, were used. Furthermore, other multimodal datasets were collected or a batch of real-world multimodal applications were constructed as test targets, representing multimodal processing scenarios of different domains and complexities, for further exploration.

[0162] Table 1 shows the test results of different fuzzing input generation methods. The baselines for comparison include traditional unimodal fuzzing tools, heuristic multimodal splicing schemes, and several variants that remove key components. The results show that the complete deep learning framework achieves higher values ​​in all three coverage metrics, with branch coverage and statement coverage of approximately 58.9% and 55.4%, respectively, while the other methods are all below half. Furthermore, the complete deep learning framework achieves a function coverage of approximately 62.7%, while also resulting in a higher number of new paths and more unique vulnerability types. Compared to unimodal tools such as AFL++, coverage and path growth show a stable leading trend. Notably, the complete framework maintains higher defect detection capabilities while having a lower false positive rate, indicating that the fusion of representations and format constraints suppresses invalid exploration while ensuring executability. Although the heuristic multimodal splicing provides some improvement over the unimodal approach, its gains are significantly weaker than the complete framework, which employs conditional generation and mutation coordination, due to the lack of joint representations and discriminative guidance. The two ablation versions, which remove feature fusion or deep learning mutations, also show varying degrees of degradation, indicating that both are necessary components for the final performance.

[0163] Table 1. Test results of different fuzzy test input generation methods

[0164] method Branch coverage / % Statement coverage / % Function coverage / % Number of new paths Unique vulnerability count High-risk vulnerabilities False alarm rate / % AFL++ (Single-modal) 41.2 ± 0.7 38.7 ± 0.5 45.0 ± 0.6 1 830 ± 42 7 ± 1 1 8.3 Heuristic multimodal stitching 47.5 ± 0.9 44.1 ± 0.8 51.3 ± 0.7 2 410 ± 55 11 ± 2 2 10.0 Ablation: Featureless fusion 51.3 ± 0.8 48.0 ± 0.6 54.6 ± 0.9 2 950 ± 61 14 ± 2 3 6.7 Ablation: No DL mutation 49.8 ± 0.6 46.5 ± 0.7 52.1 ± 0.5 2 780 ± 58 12 ± 2 2 7.1 Complete Deep Learning Framework 58.9 ± 1.1 55.4 ± 0.9 62.7 ± 1.0 3 620 ± 73 19 ± 3 5 4.5

[0165] To examine the cross-scenario adaptability of the method, task-based evaluations were conducted on various combinations, including text and image, text and code, sensor and code, text and video, and image and audio. Table 2 shows the test results of the fuzzy test input generation method in different scenarios. In scenarios such as code generation assistant and autonomous driving perception, the 24-hour values ​​of the target metric increased by approximately 19.3% and 26.5% respectively compared to the initial values, indicating that the generation and mutation strategies can continuously produce effective test inputs for the target system, driving the increase in pass rate and recall rate. In the cross-modal attack task, the attack success rate increased from zero to nearly half, demonstrating that the proposed conditional generation and directed mutation can significantly expand the reach of anomalous samples. In the OCR robustness stress test, the character-level accuracy showed a significant decrease, revealing the vulnerability of the system under specific cross-modal combinations, providing direct reference value for security hardening in real-world deployments. The BLEU metric for the video description task also showed positive improvement, corroborating the method's ability to expand coverage of multimodal sequence-based applications over a long time budget.

[0166] Table 2. Test results of fuzzy test input generation methods in different scenarios.

[0167] Scene Dataset Modal Combination measure initial value 24 hours later Improvement / Δ OCR robustness OCR-Service-Bin Text + Image Character-level accuracy ↓ 94.1 % 67.3 % -26.8 Code generation assistant CodeSearchNet-Py Text + Code Generate function pass rate ↑ 62.4 % 81.7 % +19.3 Autonomous driving perception Industrial-PLC-IoT Sensors + Code Anomaly detection recall rate ↑ 58.7 % 85.2 % +26.5 Video Description Video-Caption-Tiny Text + Video BLEU-4 ↑ 29.6 35.8 +6.2 Cross-modal attacks CrossFire AudioCaps Image + Audio Attack success rate ↑ 0 % 47.5 % +47.5

[0168] Ablation experiments explain the marginal contributions of each module at the component level. Referring to Table 3, the test results of fuzzy test input generation methods for different ablation projects show that removing the multimodal fusion network reduces both coverage and unique vulnerabilities, verifying the crucial role of cross-modal dependency modeling in reaching deep semantic paths. Removing deep learning-guided mutations, while maintaining high execution speed, significantly degrades coverage and defect detection capabilities, indicating that relying solely on traditional perturbations is insufficient for stable exploration of high-value boundaries. The most significant degradation occurs when feedback stimulus is removed, demonstrating that composite scoring-driven energy scheduling is key to maintaining long-term effective exploration. In the generator replacement experiment, the variant using a variational autoencoder shows a slight decrease in coverage but faster execution speed per unit time, reflecting the trade-off between exploration intensity and throughput in different generation architectures. This invention primarily uses conditional generative adversarial networks and sequence generation models, and employs a pluggable strategy to accommodate variational decoding, achieving a better balance between coverage, anomaly reach, and execution efficiency.

[0169] Table 3. Test results of fuzzy test input generation methods for different ablation projects.

[0170] ablation item Branch coverage / % The only vulnerability Average execution speed / use cases · min⁻¹ Remark Remove multimodal fusion network 51.3 (-7.6) 14 (-5) 420 splicing features only Remove DL mutants 49.8 (-9.1) 12 (-7) 480 Traditional mutation operators Remove feedback incentives 45.2 (-13.7) 9 (-10) 465 No energy dispatch Replace GAN → VAE 55.7 (-3.2) 16 (-3) 390 Generative diversity decreased slightly Complete Deep Learning Framework 58.9 19 375 Baseline

[0171] This invention presents a multimodal intelligent fuzzy testing input generation method and system based on deep learning. Through closed-loop collaboration of feature extraction and fusion, condition generation and directed mutation, and execution and feedback, it empirically improves the coverage increment, the number of unique vulnerabilities discovered, and the ability to reach high-risk defects. It also demonstrates good transferability and engineering robustness in multiple scenario tasks.

[0172] The present invention also provides an electronic device, comprising: a processor, a memory, and a computer program, wherein the computer program is stored in the memory and configured to be executed by the processor, and the computer program, when executed by the processor, implements the steps of the deep learning-based multimodal intelligent fuzz test input generation method as described above.

[0173] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the deep learning-based multimodal intelligent fuzz test input generation method as described above.

[0174] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep learning-based multimodal intelligent fuzz test input generation method as described above.

[0175] This invention is described with reference to a flowchart of a method according to embodiments of the invention. It should be understood that each step in the flowchart and combinations thereof can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose or special-purpose device, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the device, embedded processor, or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 A component that specifies a function in one or more processes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that enables a boot device, embedded processor, or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction components, which are implemented in a process Figure 1 The function specified in one or more processes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

Claims

1. A method for generating multimodal intelligent fuzzy test input based on deep learning, characterized in that, Includes the following steps: S1. For multimodal inputs containing text, images, audio, and code, pre-trained models and abstract syntax trees are used to extract unimodal features respectively. Then, a fusion network based on attention mechanism is used to align and model the unimodal features under different modalities to obtain a unified joint representation. S2. The deep generative model is driven by the joint representation to generate candidate inputs, and the mutation operator is used in the latent space to perform targeted mutation to improve the coverage increment and anomaly triggering capability. At the same time, the format and semantic constraint verification is performed, and a high-value seed pool is maintained. The high-value seed is a sample that meets the constraints of the generative model and has a return exceeding a preset threshold. S3. Inject candidate inputs into a controlled sandbox environment, collect statement coverage, branch coverage, path coverage, and resource and log operation metrics through instrumentation, construct a composite feedback score to prioritize samples and schedule computational budgets, and write high-value samples that meet specified conditions back to the seed pool to form a closed-loop optimization.

2. The method according to claim 1, characterized in that, In step S1, for the image modality, a pre-trained convolutional neural network is used to extract the visual features of the image; for the text modality, a pre-trained language model is used to obtain the context-aware word vectors or sentence vector representations of the text; for the audio modality, an audio feature extraction model is used. For code modalities, abstract syntax trees are used to extract the structural and semantic features of the code; Let the unimodal representations of text, image, audio, and code be as follows: Where m represents the length of the text modality input sequence, n represents the number of visual feature vectors obtained after the image modality is processed by the convolutional neural network, p represents the number of time frames or feature segments obtained after feature extraction of the audio modality, and q represents the number of syntax tree nodes or code structural units of the code modality. , , and These represent the original dimensions of text, image, audio, and code modal features, respectively. We obtain this through linear mapping: , ,in, , , and These represent linear mapping matrices for text, image, audio, and code modalities, respectively, used to map vectors from their original dimensions to a unified dimension d; Building features , Encode position and modality type, and... Input to a cross-modal encoder to obtain the output The joint characterization is obtained through convergence. .

3. The method according to claim 1, characterized in that, In step S2, the deep generative model includes a condition generator. With discriminator The generator learns to output samples that are reasonable but may reach boundary behaviors across various modalities. The discriminator distinguishes between real and generated samples and can be extended to predict anomalous tendencies. Through adversarial or variational training, a diverse and effective candidate set is formed, satisfying the following: ； ； in, To obtain a joint representation based on multimodal inputs, It is random noise. For seed library embedding vectors, For the generated candidate input, For discriminator Based on real samples and The output multi-metric vector includes fidelity. Incremental Coverage With anomaly probability measure.

4. The method according to claim 3, characterized in that, generator The specific implementation is one or a combination of a generator of a conditional generative adversarial network, a decoder of a conditional variational autoencoder, or a self-attention sequence generator with the decoder as the main component. when When using a generator with a conditional generative adversarial network, the representation J, random noise Z, and seed library embedding vector are fused. After being concatenated or linearly mapped, the data is input into the generator network, where candidate samples for each modality are generated through deconvolution, feedforward networks, or attention modules. ; when When using a conditional variational autoencoder in the decoder, the encoder first bases the decoder on the fused representation J and the seed library embedding vector. The latent variable distribution parameters are obtained, and then the latent variable vector is generated using reparameterization techniques. The decoder then performs controlled sampling and continuous interpolation in the latent space to generate cross-modal candidate samples with a coherent structure. ; when When using a decoder-centric self-attention sequence generator, the fused representation J is used as the conditional context, along with a random perturbation vector or a seed embedding vector. Cross-modal sequences or structured data are generated through a multi-head self-attention mechanism, and candidate samples are obtained through stepwise sampling or autoregressive generation. ; When using a combination approach, The three types of generation architectures generate their respective candidate sample fragments, and the results of the three are aggregated in the output stage to form a unified multimodal candidate sample set.

5. The method according to claim 1, characterized in that, In step S2, the operators for directional mutation include one or more of the following: gradient ascent search on the potential vector or output representation, parameter perturbation, masking and time shifting, amplitude perturbation, structural transformation based on abstract syntax tree, and equivalent interface replacement. The operators are selected probabilistically weighted based on historical returns to balance exploration intensity and effectiveness. Establish a set of mutation operators for each mode. The operator sampling probability is given by the following formula: ； Among them, historical returns By operator The composite feedback signals obtained in the previous several rounds of generation-execution-feedback loops are accumulated or weighted averaged to obtain the result. The threshold value is used.

6. The method according to claim 1, characterized in that, In step S2, the format and semantic constraint verification includes verifying the legality of the input structure, the range of values, and the temporal dependencies, and triggering lightweight repair when the constraints are not met so that the candidate input can be parsed and executed by the system under test.

7. The method according to claim 1, characterized in that, In step S3, the composite feedback score is calculated using the following formula: ； in, Indicates the sample Composite feedback score, Indicates execution sample The weighted coverage increment, , For collaborative coverage weighted matrix, For the sample A single execution yields a coverage metric: , To collect collaborative coverage metrics on the multimodal model side: , For augmenting the coverage vector: , To cover the baseline, These represent the number of statement-level, branch-level, and path-level coverages reached after executing sample x. The branch-level coverage reflects the hit rate of program conditional branches, while the path-level coverage measures the exploration depth and diversity of program control flow paths. These represent the fusion layer hit rate, attention head activity rate, and key operator activation rate, respectively. For resource vectors, As an indicator of novelty or rarity, The consistency conflict intensity is derived from the breach metric of cross-modal matching and format validation; This represents the probability of an anomaly. For weight parameters; based on Perform priority and budget scheduling and update the seed pool.

8. The method according to claim 7, characterized in that, in accordance with When performing priority and budget scheduling, the scheduling probability is: Where T is the temperature parameter to balance exploration and utilization; When the sample satisfies the constraint verification function and When the score exceeds a preset threshold, the seed pool is updated. ; and will and This signal is sent back to the upstream strategy as a reward. This indicates that a deduplication operation is performed on the candidate sample set.

9. A multimodal intelligent fuzzy test input generation system based on deep learning, characterized in that, include: The feature extraction and fusion submodule is used to extract single-modal features from multimodal inputs containing text, images, audio and code using pre-trained models and abstract syntax trees respectively, and to align and model the single-modal features under different modalities through a fusion network based on an attention mechanism to obtain a unified joint representation. The multimodal input generation and mutation submodule is used to drive the deep generative model to generate candidate inputs based on the joint representation as a condition, and to implement targeted mutation using mutation operators in the latent space to improve coverage increment and anomaly triggering capability. At the same time, it performs format and semantic constraint verification and maintains a high-value seed pool, wherein the high-value seeds are samples that meet the constraints of the generative model and have a return exceeding a preset threshold. The fuzz test execution and feedback submodule is used to inject candidate inputs into a controlled sandbox environment. It collects statement coverage, branch coverage, path coverage, and resource and log runtime metrics through instrumentation, constructs a composite feedback score for sample priority and computational budget scheduling, and writes high-value samples that meet specified conditions back to the seed pool to form a closed-loop optimization.

10. An electronic device, characterized in that, include: processor; Memory; And a computer program, which is stored in the memory and configured to be executed by the processor, wherein the computer program, when executed by the processor, implements the steps of the deep learning-based multimodal intelligent fuzz test input generation method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the deep learning-based multimodal intelligent fuzz test input generation method as described in any one of claims 1-8.

Images