Defect severity prediction method based on multi-modal contrast learning

Abstract

The invention requests to protect a defect severity prediction method based on multi-modal comparative learning, which comprises the following steps of: firstly, performing standardization processing on a defect report through Prompt and a guide large language model, generating explainable defect level description, and then extracting semantic features by using a text encoder of CLIP. Secondly, constructing a symbol-level hypergraph structure corresponding to the source code, and designing an improved hypergraph neural network to capture an advanced structure dependency relationship in the source code so as to obtain structural modal features; then, source codes related to the defects are visualized into images, and visual semantic information of the source codes is extracted through a CLIP visual encoder; and finally, fusing the structure representation of the code with the visual features, carrying out comparative learning alignment with text modal features, modeling multi-modal association through a shared semantic space, and realizing accurate prediction of the severity level of the defect report. According to the method, the intelligence level and accuracy of defect severity prediction are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of software testing, and particularly relates to a defect severity prediction method based on multi-modal contrast learning. BACKGROUND

[0002] Software defect prediction plays a crucial role in the software development life cycle, helping developers identify code defects early, thus avoiding wasting time and effort. Traditional software defect detection methods, such as manual testing and code inspection, are effective but often require significant human and time costs. In contrast, if potential defect modules can be predicted early in development through automated means, it will help developers implement targeted optimization at a lower cost, thus significantly improving code quality. In this context, software defect prediction (SDP) has become a key technology for ensuring software quality as it can automatically identify defect modules, and has received much attention in the field of software engineering and testing in recent years.

[0003] The core challenge of software defect prediction is how to accurately identify high-risk defect modules in the code that truly affect system stability. Although deep learning and other classification algorithms have been widely used in software defect prediction, existing methods generally have a fundamental limitation: they assume that all defects have equal impact on the system. However, in actual development, the severity of different defects varies significantly. For example, defect severity, as a key attribute labeled by software developers, directly reflects the destructive power of defects on system operation and is an important basis for determining repair priorities. However, current methods mostly ignore the semantic information (such as natural language descriptions) in defect reports, making it difficult to classify defects from a business perspective and thus weakening the actual guiding value of the prediction results. In addition, deep learning and other methods require a large amount of labeled training data to achieve satisfactory results, and large-scale labeling of software defects and defect reports often requires a large number of professional software developers, which will increase the human cost of developing software for enterprises.

[0004] Defect report severity prediction is an important research direction in the field of software engineering, aiming to analyze defect report content through automated technology and predict its severity level (such as Critical, Major, Minor, etc.), thus helping development teams determine software defect repair priority. This technology is of great significance to improving software maintenance efficiency and reducing operation and maintenance costs in the field of software engineering. Existing technical solutions mainly revolve around machine learning and deep learning, which can be summarized into the following two directions:

[0005] Traditional machine learning methods usually adopt the idea of combining feature engineering and classification algorithms. First, structured features are extracted from defect reports, including text features (such as TF-IDF), metadata features (such as reporters, components, historical repair times), etc., and then a support vector machine, random forest, or other classifier is used for modeling. The advantage of this method is that the model is highly interpretable and suitable for small-scale data sets. However, its performance is heavily dependent on manual feature design, making it difficult to capture complex semantic information in defect reports.

[0006] Deep learning methods overcome the limitations of manually designed features through end-to-end training. Typical technical solutions include: 1) CNN / LSTM-based sequence models that can handle text sequence features of defect reports; 2) pre-trained language models (such as BERT, RoBERTa) that use large-scale pre-trained models to obtain better semantic representations. The advantage of deep learning is its strong end-to-end automatic feature extraction capability, but it still requires a large amount of labeled data to train a satisfactory model.

[0007] The core of the CLIP (Contrastive Language-Image Pre-training) model is to use the Internet's vast image-text paired data (about 400 million pairs) to build a unified cross-modal semantic understanding system using a self-supervised learning mechanism. The CLIP model specifically uses a dual-encoder design: the visual side uses advanced networks such as Vision Transformer or ResNet to extract image features, and the language side uses a Transformer architecture to analyze text semantics. Through a carefully designed contrastive loss function, the model aligns the features of image-text pairs in the vector space - bringing closer the features of semantically matching image-text pairs, while pushing away unrelated features. This training method enables CLIP to have unprecedented cross-modal understanding capabilities, most notably supporting "zero-shot" reasoning: only the category text description is needed, without any fine-tuning, to complete image classification tasks.

[0008] Existing technologies still face the following key challenges in defect severity prediction:

[0009] (1) Strong dependence on labeled data: existing methods generally rely on a large number of defect reports with accurately labeled severity levels for training, which often involves high costs and low efficiency in obtaining high-quality labeled data;

[0010] (2) Single modality information: current defect reports are mainly based on subjective text descriptions by testers or developers, which makes it difficult to fully and objectively reflect the true severity of defects;

[0011] (3) Limited structural expression capability: traditional graph models mainly model the static structural relationships of class-level or method-level code, making it difficult to effectively represent the complex control and data dependencies within the code

[0012] hierarchical relationship.

[0013] After searching, application publication number CN118864362A, a medical image prediction method based on multi-modal contrast learning, relates to the field of image processing, which includes the following steps: obtaining a lung nodule image dataset and pairing text embedding data for the image dataset; based on the pre-processed lung nodule image dataset and text embedding data, a multi-modal contrast learning model is constructed, and the multi-modal contrast learning model is initially trained; adjust the multi-modal contrast learning model parameters after the initial training and perform secondary training, and sequentially perform performance testing and performance evaluation on the trained multi-modal contrast learning model; use the multi-modal contrast learning model that meets the performance evaluation standard to predict the lung nodule image dataset, and obtain the benign and malignant prediction results of the lung nodule.

[0014] Compared with the disclosed patent, our solution has significant improvements in application scenarios, modal design, model structure, and interpretability. The disclosed patent mainly faces the medical image diagnosis task, and predicts the benign and malignant of lung nodules through the contrast learning of medical images and text embedding. Although its technical idea can utilize the complementary information of images and texts, it still has two obvious shortcomings: one is the single modality type, which cannot capture complex structured information; the second is the weak interpretability of the contrast learning results, the prediction process is more like a "black box", which is difficult to support the transparency of decision-making. To solve these problems, our patent proposes a defect severity prediction method based on multi-modal contrast learning, which expands the application scenario from medicine to software engineering, and is more universal and practical. Specifically, we not only use large language models to standardize and semantically interpret defect reports, enhancing the interpretability of the text modality, but also construct a symbol-level hypergraph structure of the source code and design an improved hypergraph neural network to capture high-order dependency relationships, thus making up for the lack of the disclosed patent in modeling complex structured information. In addition, we also visualize the source code as an image, extract semantic features through a visual encoder, and align them with the text and structure modalities through contrast learning, realizing the deep fusion of the three modalities. This design not only solves the problems of single modality and structure deficiency, but also provides clearer semantic support in the prediction process, significantly improving the accuracy and interpretability of defect severity prediction, and reflecting higher innovation and technical value. SUMMARY

[0015] The present invention aims to solve the problems of the prior art. A defect severity prediction method based on multi-modal contrast learning is proposed. The technical solution of the present invention is as follows:

[0016] A defect severity prediction method based on multi-modal contrast learning, comprising the following steps:

[0017] the text modality feature extraction step, the code modality feature extraction step, the visual modality feature extraction step, and the multi-modal contrastive learning and classification step.

[0018] Further, the text modality feature extraction step specifically includes: applying a large language model and a Prompt technology to perform standardized processing on the defect report, converting the natural language description into a structured defect description; using a text encoder in a contrastive language-image pre-training CLIP model to convert the optimized defect text description into a semantic vector representation f text , and extracting deep semantic features thereof.

[0019] Further, the code modality feature extraction step specifically includes:

[0020] First, a symbol-level hypergraph structure is constructed to describe the high-order dependency relationship in the defect code.

[0021] Next, an improved hypergraph neural network IHNN is designed to learn and propagate information on the symbol-level hypergraph structure; in each layer of the IHNN, the node features are first aggregated to the hyperedge to which the node belongs, so as to capture the combined semantics of the symbols in the local context, and finally obtain a code segment representation vector f code Comprehensively integrates semantic information and high-order structural relationships at the symbol level.

[0022] Further, the construction of a symbol-level hypergraph structure to describe the high-order dependency relationship in the defect code specifically includes:

[0023] In the symbol-level hypergraph structure, each symbol token in the code is represented as a node, and the control flow and data dependency relationship is modeled by a hyperedge: each hyperedge connects multiple semantically related nodes to represent their joint semantics or behavioral dependency; based on the control flow graph CFG, the code segments with control structures such as conditions, branches, and loops in the program are identified, and multiple tokens with control dependency in these structures are collectively constructed as a control flow hyperedge to reflect their joint influence on program path selection; based on the data dependency graph DDG, a set of variables with definition-use or transmission dependency is identified, and all tokens participating in the same data flow process are constructed as a data dependency hyperedge to reflect their semantic coupling;

[0024] The hypergraph constructed in the above manner captures the high-order relationship between multiple symbol tokens in the defect code, and obtains a complete and structurally rich symbol token-level hypergraph representation, providing a structurally complete and semantically rich input representation for the subsequent hypergraph neural network;

[0025] Let G=(V, E, W) be a hypergraph with N vertices and M hyperedges, where V, E are the node set and hyperedge set respectively; a node v∈V is composed of a symbol token, and a hyperedge e∈E is composed of a set of nodes composed of a control flow graph and a data dependency graph; denotes the i-th node in the hyperedge e, and |e| denotes the number of nodes in the hyperedge e, denotes the node feature matrix, x v denotes the feature vector of node v; the initial representation of the node is obtained by calculating the characters by Word2vec; each hyperedge e∈E is assigned a positive weight W ee , and stored in a diagonal matrix ; the point-edge association matrix of the hypergraph is denoted as G=(V, E, W); the association matrix is defined as follows,

[0026] The vertex degree is defined as H ie denotes the association value of the i-th node and the hyperedge e. The degree of the hyperedge is defined as The degree matrix and are diagonal matrices; the weight of the hyperedge e m is defined as follows,

[0027]

[0028] where δ i is the kernel parameter, taking the average distance measure between symbols, x i and x j are the feature vectors of the corresponding nodes in the same hyperedge.

[0029] Further, the improved hypergraph neural network IHNN is designed to learn and propagate information on the symbol-level hypergraph structure, specifically including: in each layer of IHNN, first aggregate the node features to the hyperedge to which the node belongs, i.e. node features X→hyperedge features P, to capture the combined semantics of each symbol in the local context:

[0030]

[0031] where σ(·) is a commonly used nonlinear activation function in deep models, x (0) denotes the initial feature matrix, denotes the node set related to the hyperedge e, and α ie is the weighting coefficient of node i to the hyperedge e feature, determined by the relationship strength between the node and the hyperedge or the similarity of the node features, denotes the feature representation vector of the i-th node in the l-th layer of the network. is the first layer hyperedge feature weight matrix of network learning; the features of the hyperedge are propagated back to the associated nodes, i.e., hyperedge feature→node feature, to realize cross-structure semantic information enhancement;

[0032]

[0033] wherein X (l+1) represents the feature representation matrix of all vertices in the l+1th layer of the network, E i represents a set of all hyperedges associated with the ith vertex, represents the feature of the hyperedge e in the lth layer, and the matrix Θ (l) and is the lth layer parameter in the learning of the node feature of the hypergraph neural network; after updating the node feature, the hyperedge weight is recalculated through formula (2) to realize dynamic learning of the hypergraph;

[0034] Through the stacking of multiple layers of IHNN, more distant semantic propagation and context modeling between nodes can be realized, so that the dependency relationship between nodes is fully captured, and the propagation across nodes and hyperedges further enhances the context information of the code, thereby obtaining a structural perception, semantically complete symbolic level feature representation; after the output of the HGNN network, the node features inside each hyperedge are aggregated to obtain the corresponding hyperedge feature Then, the attention mechanism is used to weight and sum all hyperedge embeddings, i.e., to obtain the attention weight of the hyperedge e, wherein a and are learnable attention parameters ， so that the model can automatically focus on the semantic structure or key sentence area; the feature calculation formula of the defect code segment is as follows:

[0035] f code =∑ e∈E α e p e (5)

[0036] The finally obtained code segment representation vector f code comprehensively integrates the semantic information and high-order structure relationship at the symbolic level.

[0037] Further, the visual modality feature extraction step specifically includes:

[0038] The source code corresponding to the defect report is visualized and converted into a high-quality image form to enhance its readability and semantic expression, and then input into the visual encoder of the CLIP model to extract the visual modality feature representation f image, a four-layer network structure composed of convolutional layers and max-pooling layers is designed to gradually extract low-level to middle-level semantic features of the code image. The visual processing method can capture code semantics and structure information, and in combination with the defect report text modal feature, the defect report severity prediction can be realized under the CLIP structure.

[0039] Further, the multi-modal contrast learning and classification step specifically includes:

[0040] First, the visual features are taken as queries to calculate the relevance with the structure features to obtain a weighted representation of the structure features, and the specific calculation process is as follows:

[0041]

[0042] where W Q is a learnable weight matrix for query; Atten image→code represents the attention weight of visual features and structure features, f image represents the visual modal feature of the defect code segment, f code represents the defect code segment feature vector learned by IHNN. This process will generate a weighted structure feature representation for each visual feature, representing the association between the visual feature and the structure feature;

[0043] Then, the structure features are taken as queries to calculate the relevance with the visual features to obtain a weighted representation of the visual features, and the calculation process is as follows:

[0044] The structure features are weighted by the attention relationship between the structure features and the visual features to generate a weighted representation from structure to visual; after the feature interaction from visual to structure and structure to visual through the above two attention processes, the results of the two can be fused to obtain the final fusion feature:

[0045] f fused =βAtten image→code +(1-β)Atten code→image (8)

[0046] Where β is a parameter learned through model training;

[0047] Finally, the fused defect code feature f fused and the defect report text feature f text are input into the contrast loss to optimize the semantic distance between different severity samples, so that the model can distinguish the defect level in the same feature space; during the training process, the model parameters are optimized by minimizing the contrast loss, so that the model can effectively capture the deep semantic relationship between the defect code and the text description; the total loss function of the framework is:

[0048]

[0049]

[0050] where and is the feature vector of the k-th sample, sim(·,·) is the similarity measure, τ is the temperature coefficient that controls the degree of contrast of similarity, y k is the label of the k-th sample in the training phase, is the class prediction probability based on the code fusion features and text features.

[0051] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the multi-modal contrast learning-based defect severity prediction method when executing the program.

[0052] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the multi-modal contrast learning-based defect severity prediction method.

[0053] The advantages and benefits of the present application are as follows:

[0054] (1) The present application proposes an unsupervised defect severity prediction method based on multi-modal contrast learning: the present application innovatively introduces a multi-modal contrast learning framework, which realizes unsupervised prediction of software defect severity level by fusing text, image and structure information, and significantly reduces the dependence on a large amount of manually annotated data. This method breaks through the dependence on high-cost labeled data in traditional supervised learning, and can effectively utilize large-scale unlabeled data for training to enhance the adaptability and generalization ability of the model.

[0055] (2) A multi-modal defect severity prediction framework is constructed, which fuses text, visual and structure information:

[0056] This framework improves the comprehensive perception and understanding ability of the model for defect information through multi-modal feature fusion. Specifically, the framework simultaneously utilizes the text information of the defect report, the structure information of the defect code and the code visual information related to the defect to form a multi-dimensional code knowledge representation, which comprehensively captures the potential defect features in the source code. Through contrast learning, the features of different modalities are aligned in a unified semantic space, thereby improving the accurate judgment of the model for defect severity.

[0057] (3) A hypergraph neural network model is designed for modeling the complexity control and dependency relationship of the defect code:

[0058] The present proposal further designs a token hypergraph structure and uses an improved hypergraph neural network (IHNN) for feature learning, which can effectively model the complex control flow and data dependency in defect code. Unlike traditional graph neural networks, hypergraph neural networks can handle high-order multivariate associations between nodes, significantly enhancing the expressiveness of token-level code. This innovative design can improve the expressiveness and semantic accuracy of code information,

[0059] thereby providing a more effective basis model for defect positioning and severity prediction.

[0060] The core innovation and technical implementation of this scheme provide a new idea and technical path for the field of defect severity prediction, with high practical value and wide application prospects.

[0061] (1) Existing methods generally rely on a large number of defect reports with accurately labeled severity levels for training. This method not only requires a large amount of manual annotation data, but the quality and consistency of the annotation will also directly affect the performance of the model. However, the present application first proposes an unsupervised defect severity prediction method based on multi-modal contrastive learning, which effectively alleviates the dependence on labeled data and significantly reduces the annotation cost and time consumption. By using a large amount of unlabeled data, the model can be trained under unsupervised conditions to automatically capture the severity of defects. Through the contrastive learning framework, the model can learn from text, visual, and structural features and accurately judge the defects, thereby improving the adaptability and generalization ability of the model. This method not only solves the problem of lack of labeled data, but also enhances the robustness of the model in different software scenarios.

[0062] (2) The content of the current defect report text often cannot fully and objectively reflect the true severity level of the defect, mainly due to the subjectivity and incompleteness of the text description. Traditional methods usually rely only on text features for severity prediction, ignoring the structural information of the source code and the visual representation of the code. The present application first constructs a multi-modal defect severity prediction framework that integrates text, visual, and structural information, which can comprehensively consider information from different modalities to fully perceive and understand defect code. Specifically, the text modality provides natural language descriptions of defects, the visual modality provides visual representations of code, and the structural information helps the model understand the control flow and data dependency of the code. Through this multi-modal feature fusion, the model can comprehensively evaluate defects at different levels, improving the accuracy and reliability of defect severity prediction.

[0063] (3) Traditional graph models usually only model the static structural relationships of class-level or method-level code. These methods are difficult to effectively represent the complex control flow and data dependency of token-level relationships in the code, resulting in limitations in handling complex source code defects. In contrast, the hypergraph neural network model designed in the present application can model the complex control flow and data dependency relationships in the defect code, enhancing the expression ability of the source code from the token level. The hypergraph structure can effectively capture the high-order multivariate association between multiple nodes, making up for the limitations of traditional graph models that can only handle second-order relationships. In the analysis of defect code, the hypergraph neural network can model more complex and fine-grained dependency relationships in the code, which enables the model to more accurately represent the code logic and improve the accuracy of defect positioning and severity judgment. In addition, the design of the hypergraph structure also enables the model to capture complex dependency relationships across functions and modules, thereby improving the ability to perceive defects in large and complex code bases.

[0064] (4) By organically combining the above technologies, the present application not only theoretically realizes accurate prediction of defect severity, but also improves the robustness of the model to different types of defects in practical applications. Whether facing scarce labeled data, complex code structure, or lack of comprehensive text description of defect reports, this method can fully mine and fuse multi-modal information and efficiently train in an unsupervised environment. This cross-modal learning mechanism significantly improves the model's ability to distinguish defect severity, especially when faced with diverse source code and report content, the model can maintain high accuracy and stability. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 is a flowchart of the defect severity prediction method based on multi-modal contrast learning provided by the present application;

[0066] Figure 2 is a prompt design schematic diagram;

[0067] Figure 3 is a flowchart of the token-level improved hypergraph neural network method;

[0068] Figure 4 is the visual encoder network structure of the present framework. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the drawings in the embodiments of the present application. The described embodiments are only a part of the embodiments of the present application.

[0070]

[0071] The technical problems to be solved in the present scheme include:

[0072] (1) Develop an unsupervised defect severity prediction method based on multi-modal contrast learning to effectively alleviate the dependence on labeled data;

[0073] (2) Construct a multi-modal defect severity prediction framework that integrates text, visual and structural information to improve the comprehensive perception ability of the model for defect information;

[0074] (3) Design a hypergraph neural network model for modeling defect code complexity control and dependency relationships to enhance the source code expression ability at the token level.

[0075] The present application proposes a defect severity prediction method based on multi-modal contrast learning, aiming to integrate the text information of defect reports, the visual features of source code and the structural features, and realize the automatic prediction of unlabeled defect severity. The method framework consists of four key modules: text modal feature extraction, visual modal feature extraction, code structure modal modeling, multi-modal contrast learning and classification. Specifically, first, through the careful design of Prompt, guide the large language model (LLM) to standardize the processing of defect reports, and generate interpretable defect level description, then use the text encoder of CLIP to extract semantic features. Second, construct the corresponding symbol-level hypergraph structure of the source code, and design an improved hypergraph neural network to capture the advanced structural dependency relationships in the source code, so as to obtain the structural modal features. Then, the source code related to the defect is visualized as an image, and the visual semantic information of the image is extracted by using the visual encoder of CLIP. Finally, the structure representation of the code is fused with the visual features, and the contrast learning alignment is performed with the text modal features, the multi-modal association is modeled through the shared semantic space, and the accurate prediction of the severity level of the defect report is realized. This method fully excavates the multi-source information related to defects, and significantly improves the intelligent level and accuracy of defect severity prediction under the condition of lacking labeled data. The overall process of the present application is shown in Figure 1 .

[0076] (⊙ represents inner product, L CE cross-entropy loss, L fused-text contrast loss)

[0077] 1. Text modal feature extraction

[0078] In the software development process, defect reports are usually written by developers or testers in natural language, describing the problems and their impact on the corresponding code segment. However, these descriptions often have strong subjectivity, ambiguous language, and non-standard structure, which seriously affect the accurate judgment of defect severity. This proposal uses large language models and Prompt technology to rewrite and standardize the text of defect reports, making their content clearer and more structured, and enhancing the understanding of defect severity information. For example, Figure 2 As shown in the first step, the proposed Prompt can eliminate language noise, missing fields, and ambiguous expressions, resulting in a high-quality defect report that enhances key information.

[0079] Traditional natural language processing methods for defect severity prediction often rely on rules or shallow features, making it difficult to understand the implicit semantic information and potential severity clues in defect reports. Large language models (LLM) have strong language understanding and reasoning capabilities, allowing them to more accurately analyze the content of defect reports and implement integrated processing of "classification + explanation." For example, Figure 2 As shown in the second step, the proposed Prompt, combined with the powerful reasoning ability of LLM, automatically analyzes the severity of defect reports, outputs structured labels and reasons, and can be used for training data labeling and multi-modal text-based corpus. This method can significantly improve the interpretability of defect report severity, reduce the reliance on manual annotation, and provide strong support for building an efficient and intelligent defect report analysis system.

[0080] Finally, the proposed CLIP model uses a text encoder (Text Encoder) to convert the optimized defect text description into a semantic vector representation (i.e., f text ), extracting its deep semantic features for subsequent multi-modal feature alignment. This module realizes the full-process optimization of defect reports from unstructured to structured, from language confusion to semantic consistency, and from text description to category semantic vector.

[0081] 2. Code modal feature extraction

[0082] In defect severity prediction, relying solely on natural language descriptions of defect reports cannot fully express the actual severity level of defects, especially in cases where the code structure is complex or the defect is implicit. Defect reports often involve specific code modules, so combining source code information can help to better understand the potential severity of defects. However, traditional code feature extraction methods are often based on function-level or method-level structure representation, making it difficult to fully capture the complex control and data dependencies at the token level within the code.

[0083] Therefore, the present proposal proposes a token-level hypergraph modeling and semantic representation method for defect code segments, aiming to fully exploit the structural and semantic dependency information in source code, such as Figure 3 As shown. First, by parsing the target defect code segment through Abstract Syntax Tree (AST), all syntax tokens are extracted, including variables, operators, keywords, constants, etc., to construct the basic semantic unit representation, thereby serving as the nodes of the hypergraph, v∈V. Further, Control Flow Graph (CFG) and Data Dependence Graph (DDG) are used as structural prior information to construct high-order relationships in the hypergraph, i.e., hyperedges, e∈E.

[0084] Control flow connects tokens with execution order, reflecting the logical structure of program execution paths. Data dependency connects tokens with definition-use and variable transmission dependencies, embodying the semantic coupling and context dependency between variables. In actual source code, control flow and data dependency often involve multiple semantically related tokens. For example, the control path of a conditional or loop statement may depend on the joint action of multiple variables and operators. Similarly, the value of a variable may be influenced by multiple definition and use locations. Therefore, it is crucial to construct a graph structure that can model the high-order dependency relationships between multiple variables and nodes for defect code segments.

[0085] The present proposal first constructs a token-level hypergraph structure to describe the high-order dependency relationships in defect code. In this structure, each token in the code can be represented as a node, and control flow and data dependency are modeled through hyperedges: each hyperedge connects multiple semantically related nodes to represent their joint semantics or behavioral dependencies. Based on the Control Flow Graph (CFG), code segments with conditional, branching, and loop control structures are identified, and multiple tokens with control dependency relationships in these structures are collectively constructed into a control flow hyperedge, reflecting their joint influence on program path selection. Based on the Data Dependence Graph (DDG), variable sets with definition-use or transmission dependency relationships are identified, and all tokens participating in the same data flow process are constructed into a data dependency hyperedge, embodying their semantic coupling. If the defect code segment contains functions, classes, and other modules, high-order hyperedges such as function calls, class levels, and scope blocks can be further constructed to enhance the model's understanding of code semantics and structure.

[0086] The hypergraph constructed in the above manner can fully capture the high-order relationship between multiple tokens in the defect code, obtain a complete and structurally rich token-level hypergraph representation, provide a structurally complete and semantically rich input representation for a subsequent hypergraph neural network, and thus provide stronger code semantic modeling capability for defect report severity, defect detection, and code understanding.

[0087] Let G=(V,E,W) be a hypergraph with N vertices and M hyperedges, where V and E are the node set and hyperedge set, respectively. A node v E V is composed of tokens, and a hyperedge e E E is a set of nodes The hypergraph can be composed of a control flow graph and a data dependency graph. represents a node feature matrix, and x v represents the feature vector of node v, which can be obtained by calculating the characters by Word2vec to obtain the initial representation of the node. Each hyperedge e E E is assigned a positive weight W ee and is stored in a diagonal matrix The point-edge association matrix in the hypergraph represents G=(V,E,W). The association matrix is defined as follows,

[0088] The vertex degree is defined as The degree of the hyperedge is defined as Note that the degree matrix and are diagonal matrices. The weight of the hyperedge e m is defined as follows,

[0089]

[0090] where δ i is the kernel parameter, which can be the average distance measure between tokens, x i and x j are the feature vectors of the corresponding vertices in the same hyperedge. In the hypergraph neural network of the present application, the hyperedge with a larger weight should have a greater confidence in node message passing, so that the model can capture more critical structural relationships in the defect code.

[0091] Next, an improved hypergraph neural network (IHNN) is designed to learn and propagate information on the token-level hypergraph structure. In each layer of the IHNN, the node features are first aggregated to the hyperedge to which they belong (i.e., node features X→hyperedge features P) to capture the combined semantics of the tokens in the local context:

[0092]

[0093] where σ(·) is a commonly used nonlinear activation function in deep models, x (0) represents the initial feature matrix, represents the set of nodes associated with the hyperedge e, and ie is the weighted coefficient of node i to the hyperedge e feature, which is usually determined by the relationship strength of nodes and hyperedges or the similarity of node features, is the l-th layer hyperedge feature weight matrix learned by the network. Then the hyperedge features are propagated back to its associated nodes (i.e., hyperedge feature→node feature), realizing cross-structure semantic information enhancement.

[0094]

[0095] where matrix (l) and is the l-th layer parameter in the hypergraph neural network learning node feature. After updating the node feature, the hyperedge weight is recalculated by formula (2) to realize the dynamic learning of the hypergraph.

[0096] Through the stacking of multiple layers of IHNN, semantic propagation and context modeling between nodes at a longer distance can be realized, so that the dependency relationship between nodes is fully captured, and the propagation across nodes and hyperedges further enhances the context information of the code, thereby obtaining a structural perception, semantically complete symbolic level feature representation. After the output of the HGNN network, the node features inside each hyperedge are aggregated to obtain the corresponding hyperedge feature Then the attention mechanism is used to weight sum all hyperedge embeddings, i.e. get the attention weight of hyperedge e, where a and are learnable attention parameters. This allows the model to automatically focus on semantic structures or key sentence regions that have a greater impact on defect determination. The defect code segment feature calculation formula is as follows:

[0097] f code =∑ e∈E α e p e (5)

[0098] The final obtained code segment representation vector f code comprehensively integrates the semantic information and high-order structure relationship at the symbolic level, improving the model's perception and discrimination ability for complex code defects, so as to be used for subsequent defect severity prediction tasks.

[0099] 3. Visual modality feature extraction

[0100] In software defect prediction, traditional methods rely on manually designed features, which are then input into machine learning models for classification. However, these features often fail to adequately represent the deep semantic structure of code, especially the semantic information hidden in the Abstract Syntax Tree (AST). To overcome this limitation, inspired by the success of deep learning in image processing, researchers have proposed visualizing source code as images and using image classification models for defect prediction. This approach generates structured code images by mapping code characters to pixel colors, preserving the semantic and structural information of the code. Additionally, it leverages the powerful feature extraction capabilities of Convolutional Neural Networks (CNN) and other visual models to automatically learn hidden patterns from images. Through visual comparison, the structural and semantic differences between programs are more easily identified, providing a more direct and effective semantic representation for code feature learning.

[0101] The proposed solution visualizes the source code corresponding to the defect report, converting the code into high-quality image form to enhance its readability and semantic expression. The visual modal feature representation (i.e., f image ) is then extracted from the Visual Encoder of the CLIP model. The traditional CLIP uses deep ViT or ResNet as the visual encoder, which is powerful but computationally expensive. However, software code images have strong structural regularity and low visual complexity, so they do not require deep networks to extract effective semantic features. After verification, the proposed solution designs a four-layer network structure composed of convolutional layers and max-pooling layers to gradually extract low-level to mid-level semantic features of code images, as shown in Figure 4 Compared to traditional code text feature representation, this visual processing method better captures code semantic and structural information, which, combined with the aforementioned defect report text modal features, can achieve defect report severity prediction under the CLIP structure.

[0102] 4. Multi-modal contrastive learning and classification

[0103] In the defect report severity prediction task, single-modal features often fail to adequately represent the complexity of software defects. Defect report texts provide linguistic descriptions of problems, but they are ambiguous and subjective, while source code contains structural and contextual information about defects. By visualizing source code and extracting visual features, the proposed solution can identify semantic patterns using deep models, and by modeling high-order structural relationships in code using symbolic-level hypergraph neural networks, it can uncover richer syntax and control semantics in defective code. However, these two features focus on different dimensions of information in the code, and relying on only one of them cannot fully represent the behavior of the code. Therefore, how to effectively fuse code visual features (f image ∈R D and code structural features (f code ∈RD ) is the key to accurately characterize the defect code complex information. To this end, the present proposal proposes a cross-attention mechanism to effectively fuse visual and structural features, forming a unified, multi-dimensional code representation, so that the fused code features contain more complementary information and less redundant information.

[0104] First, the visual features are taken as queries to calculate the relevance with the structural features, obtaining the weighted representation of the structural features, the specific calculation process is as follows:

[0105]

[0106] Where W Q is a learnable weight matrix for query. This process will generate a weighted structural feature representation for each visual feature, representing the association between the visual feature and the structural feature.

[0107] Then, the present proposal takes the structural features as queries to calculate the relevance with the visual features, obtaining the weighted representation of the visual features, the calculation process is as follows:

[0108] The structural features are weighted by the attention relationship between them and the visual features to generate a weighted representation of the structure to the visual. Through the above two attention processes, the interaction of the features is completed from the visual to the structure and from the structure to the visual, respectively. The results of the two can be fused to obtain the final fused features:

[0109] f fused = βAtten image→code +(1-β)Atten code→image (8)

[0110] Where β is a parameter learned through model training.

[0111] Finally, the fused defect code features f fused and the defect report text features f text are input into the contrastive loss, optimizing the semantic distance between different severity samples, so that the model can distinguish defect levels in the same feature space. In the training process, by minimizing the contrastive loss, the model parameters are optimized so that they can effectively capture the deep semantic relationship between defect code and text description. The total loss function of the framework in the present proposal is:

[0112]

[0113] Where and are the feature vectors of the kth sample, sim(·,·) is the similarity measure, τ is the temperature coefficient, which controls the contrast degree of similarity, y kis a label of the kth sample in the training stage, is a class prediction probability based on the code fusion feature and the text feature. This optimization process can enhance the robustness of the model, so that the model not only performs well on the seen defect samples, but also maintains a high prediction accuracy when facing new types of defects.

[0114] The systems, apparatuses, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions.

[0115] The computer readable medium includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0116] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0117] The above embodiments should be understood as only for illustrating the present application and not for limiting the protection scope of the present application. After reading the content disclosed in the present application, those skilled in the art can make various changes or modifications to the present application, and these equivalent changes and modifications also fall within the scope defined by the claims of the present application.

Claims

1. A defect severity prediction method based on multimodal contrastive learning, characterized in that, Includes the following steps: Steps for extracting text modal features; steps for extracting code modal features; steps for extracting visual modal features; steps for multimodal contrastive learning and classification.

2. The defect severity prediction method based on multimodal contrastive learning according to claim 1, characterized in that, The text modality feature extraction step specifically includes: applying a large language model and Prompt technology to standardize the defect report, converting the natural language description into a structured defect description; and using the text encoder in the contrastive language-image pre-trained CLIP model to convert the optimized defect text description into a semantic vector representation, i.e., f. text Extract its deep semantic features.

3. The defect severity prediction method based on multimodal contrastive learning according to claim 1, characterized in that, The code modality feature extraction steps specifically include: First, a symbolic hypergraph structure is constructed to describe the higher-order dependencies in the defective code; Next, an improved hypergraph neural network (IHNN) is designed to learn and propagate information from this symbolic hypergraph structure. In each layer of the IHNN, node features are first aggregated to their respective hyperedges to capture the combined semantics of symbols in the local context, ultimately obtaining the code segment representation vector f. code It fully integrates semantic information at the symbolic level and higher-order structural relationships.

4. The defect severity prediction method based on multimodal contrastive learning according to claim 3, characterized in that, The construction of a symbolic-level hypergraph structure to describe higher-order dependencies in defective code specifically includes: In the symbolic hypergraph structure, each symbolic token in the code is represented as a node, while control flow and data dependencies are modeled through hyperedges: each hyperedge connects multiple semantically related nodes to represent their joint semantic or behavioral dependencies; based on the control flow graph (CFG), code segments with control structures including conditions, branches, and loops are identified, and multiple tokens with control dependencies in these structures are jointly constructed into a control flow hyperedge, reflecting their joint impact on program path selection; based on the data dependency graph (DDG), sets of variables with definition-use or transitive dependencies are identified, and all tokens participating in the same data flow process are constructed into a data dependency hyperedge, reflecting their semantic coupling; The hypergraph constructed in the above manner captures the high-order relationships between multiple symbol tokens in defective code, resulting in a complete and structurally rich symbol token-level hypergraph representation, which provides a structurally complete and semantically rich input representation for subsequent hypergraph neural networks; Let G = (V, E, W) be a hypergraph with N vertices and M hyperedges, where V and E are the set of nodes and the set of hyperedges, respectively; nodes v ∈ V are composed of tokens, and hyperedges... It consists of a control flow graph and a data dependency graph; Let |e| represent the i-th vertex in hyperedge e, and |e| represent the number of vertices in hyperedge e. Represents the node feature matrix, x v The feature vector of node v is represented; the initial representation of the node is obtained by Word2vec calculating the characters; each hyperedge e∈E is assigned a positive weight W. ee And stored in a diagonal matrix In a hypergraph, the vertex-edge incidence matrix is ​​used. Let G = (V, E, W); the incidence matrix is ​​defined as follows: Vertex degree is defined as H ie Let represent the association value between the i-th vertex and the hyperedge e. The degree of the hyperedge is defined as... Degree matrix and Both are diagonal matrices; hyperedge e m The weight calculation formula is defined as follows: Where δ i It is a kernel parameter, taking the average distance metric between symbols, x i and x j The eigenvectors of corresponding vertices belonging to the same hyperedge.

5. The defect severity prediction method based on multimodal contrastive learning according to claim 4, characterized in that, The design of an improved hypergraph neural network (IHNN) for learning and information propagation of symbolic hypergraph structures specifically includes: in each layer of the IHNN, firstly, aggregating node features to their respective hyperedges, i.e., node feature X → hyperedge feature P, to capture the combined semantics of various symbols in the local context. Where σ(·) is a commonly used nonlinear activation function in deep models, x (0) Represents the initial characteristic matrix. Let α represent the set of nodes associated with the hyperedge e. ie It is the weighting coefficient of node i with respect to the features of hyperedge e, which is determined by the strength of the relationship between the node and the hyperedge or the similarity of the node features. This represents the feature representation vector of the i-th vertex in the l-th layer of the network. It is the hyperedge feature weight matrix of the l-th layer of network learning; then the hyperedge features are propagated back to their associated nodes, that is, hyperedge features → node features, to achieve cross-structure semantic information enhancement; Where X (l+1) E represents the feature representation matrix of all vertices in the (l+1)th layer of the network. i Let represent the set of all hyperedges associated with the i-th vertex. The matrix Θ represents the feature of hyperedge e in the l-th layer. (l) and These are the parameters of the l-th layer in the learning node features of the hypergraph neural network; after updating the node features, the hyperedge weights are recalculated using formula (2) to achieve dynamic learning of the hypergraph; By stacking multiple layers of IHNN, semantic propagation and context modeling over longer distances between nodes can be achieved, fully capturing the dependencies between nodes. Furthermore, propagation across nodes and hyperedges further enhances the contextual information of the code, resulting in structure-aware, semantically complete symbolic feature representations. After the HGNN network outputs, the node features within each hyperedge are aggregated to obtain the corresponding hyperedge features. Then, a weighted summation of all hyperedge embeddings is performed using an attention mechanism, i.e. Obtain the attention weights of the hyperedge e, where a and It is a learnable attention parameter ， This enables the model to automatically focus on semantic structure or key statement regions; the formula for calculating defective code segment features is as follows: f code =∑ e∈E a e p e (5) The final code segment representation vector f code It fully integrates semantic information at the symbolic level and higher-order structural relationships.

6. The defect severity prediction method based on multimodal contrastive learning according to claim 1, characterized in that, The visual modality feature extraction step specifically includes: The source code corresponding to the defect report is visualized, converting it into a high-quality image format to enhance its readability and semantic expression. This image is then input into the visual encoder of the CLIP model to extract its visual modal feature representation, f. image A four-layer network structure consisting of convolutional layers and max pooling layers is designed to progressively extract low- to mid-level semantic features from code images. The visualization processing method can capture the semantic and structural information of the code. Combined with the aforementioned defect report text modal features, the severity of defect reports can be predicted under the CLIP structure.

7. The defect severity prediction method based on multimodal contrastive learning according to claim 1, characterized in that, The multimodal contrastive learning and classification steps specifically include: First, visual features are used as queries, and their relevance to structural features is calculated to obtain a weighted representation of the structural features. The specific calculation process is as follows: Among them W Q It is a learnable weight matrix used for querying; Atten image→code The attention weights f represent visual and structural features. image f represents the visual modal characteristics of the defective code segment. code This represents the feature vector of the defective code segment learned by IHNN; this process will generate a weighted structural feature representation for each visual feature, representing the association between the visual feature and the structural feature; Then, using structural features as the query, the correlation with visual features is calculated to obtain a weighted representation of the visual features. The calculation process is as follows: Structural features are weighted through the attention relationship with visual features to generate a weighted representation from structure to vision. After completing the interaction of features in both directions—visual to structural and structural to visual—through the above two attention processes, the results can be fused to obtain the final fused feature. f fused βAtten image→code +(1-β)Atten code→image (8) Where β is a parameter learned through model training; Finally, the merged defect code features f fused With the defect report text feature f text The contrastive loss is input to optimize the semantic distance between samples of different severity levels, enabling the model to distinguish defect levels within the same feature space. During training, the model parameters are optimized by minimizing the contrastive loss, allowing it to effectively capture the deep semantic relationship between defect codes and text descriptions. The overall loss function of the framework is: in and The feature vector of the k-th sample, sim(·,·) is the similarity measure, τ is the temperature coefficient, which controls the degree of similarity comparison, y k It is the label of the k-th sample during the training phase. It is a category prediction probability based on code fusion features and text features.

8. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the defect severity prediction method based on multimodal contrastive learning as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the defect severity prediction method based on multimodal contrastive learning as described in any one of claims 1 to 7.

Images