Training / application method for representation learning model, and device and medium

Abstract

Provided in the present application are a training / application method for a representation learning model, and a device and a medium. The training method comprises: extracting a plurality of source codes from an open-source repository, and on the basis of the plurality of source codes and a large language model, generating a training sample set, wherein each sample in the training sample set comprises assembly codes and natural language text; on the basis of an assembly code data set, performing pre-training to generate an assembly coder, and on the basis of a natural language data set, performing pre-training to generate a text coder; on the basis of the training sample set and a contrastive learning algorithm, performing alignment training on the assembly coder and the text coder, so as to obtain a code representation learning model and a natural language representation learning model which are semantically aligned; and on the basis of the code representation learning model and the natural language representation learning model, which are semantically aligned, constructing a representation learning model. The method of the present application significantly improves the generalization ability and accuracy of a representation learning model, and greatly reduces dependence of the model on samples and a large volume of tagged data.

Description

Training / application methods, devices, and media for representation learning models

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 2, 2024, with application number CN202410153832.0 and application name “Training / Application Method, Device and Medium for Representation Learning Model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of machine learning, and in particular to a training / application method, device, and medium for representing a learning model. Background Art

[0003] Binary code representation learning methods have high practical value in tasks such as function boundary detection, binary code search, binary code similarity detection, function prototype inference, malware classification, reverse engineering, etc.

[0004] Binary code representation learning is a method for representing binary code as continuous vectors in a vector space. Existing binary code representation learning methods are mainly divided into three categories: directly modeling raw bytes, using graphical models to establish control flow relationships between basic blocks, and representing function instructions as instruction sequences. However, the representation learning accuracy of these three methods is poor, and they all rely on a large number of training samples and annotations. They perform poorly when training samples are scarce or absent, lack sufficient generalization capabilities to handle completely unknown data types or tasks, and have poor generalization. In addition, due to the complexity and diversity of binary code and the rapid changes in programming practices, collecting and annotating large amounts of data for new tasks becomes very difficult and time-consuming, greatly increasing the cost and complexity of applying these methods to real-world problems.

[0005] Therefore, a representation learning scheme is needed that can improve the generalization ability and accuracy of the representation learning model and reduce the dependence on samples and large amounts of labeled data. Summary of the Invention

[0006] The present application provides a training / application method, device and medium for a representation learning model to solve the technical problems of existing binary code representation learning methods, such as poor generalization ability and accuracy, and dependence on samples and large amounts of labeled data.

[0007] In a first aspect, the present application provides a training method for a representation learning model, comprising:

[0008] Extracting multiple source codes from an open source repository and generating a training sample set based on the multiple source codes and a large language model, wherein each sample in the training sample set includes assembly code and natural language text;

[0009] Pre-train an assembly encoder based on an assembly code dataset, and pre-train a text encoder based on a natural language dataset.

[0010] Performing alignment training on the assembly encoder and the text encoder based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model;

[0011] A representation learning model is constructed based on the semantically aligned code representation learning model and the natural language representation learning model.

[0012] In a possible implementation, generating a training sample set based on multiple source codes and a large language model specifically includes:

[0013] For each source code,

[0014] generating different versions of binary assembly code according to the source code, and determining the natural language text corresponding to the source code using a large language model;

[0015] Generate a positive sample corresponding to the source code according to any binary assembly code corresponding to the source code and a natural language text corresponding to the source code;

[0016] Generate negative samples corresponding to the source code according to any binary assembly code corresponding to the source code and any natural language text not corresponding to the source code, wherein there are multiple negative samples and the number of negative samples is greater than the number of positive samples;

[0017] Generate a positive sample set based on the positive sample corresponding to each source code, and generate a negative sample set based on the negative sample corresponding to each source code;

[0018] A training sample set is generated according to the positive sample set and the negative sample set.

[0019] In a possible implementation, determining the natural language text corresponding to the source code using a large language model specifically includes:

[0020] Taking the pre-trained large language model as the target model, the shadow model strategy is used to train the shadow application model corresponding to the target model;

[0021] Determine a natural language text corresponding to the source code according to the shadow application model.

[0022] In one possible implementation, pre-training and generating an assembly encoder based on an assembly code dataset specifically includes:

[0023] For each assembly code data in the assembly code data set,

[0024] relocating the addresses of the assembly code data so that the assembly code data maintains a relative address relationship when processing a jump instruction;

[0025] Using a word segmentation algorithm to segment the relocated assembly code data into independent tokens to obtain code samples corresponding to the assembly code data;

[0026] A code sample set is generated according to a code sample corresponding to each assembly code data, and an assembly encoder is generated by pre-training according to the code sample set.

[0027] In a possible implementation, generating an assembly encoder based on the pre-training of the code sample set specifically includes:

[0028] Generate the initial encoder based on the Transformer model architecture;

[0029] Determining an instruction embedding corresponding to each instruction in the initial encoder;

[0030] Embedding the instructions corresponding to each instruction into the initial encoder to generate an intermediate encoder;

[0031] According to each code sample in the code sample set, the intermediate encoder is pre-trained, and the tag embedding parameters of the jump symbol and its corresponding instruction embedding are shared during the model pre-training process to obtain an assembly encoder.

[0032] In a possible implementation, performing alignment training on the assembly encoder and the text encoder based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model specifically includes:

[0033] For each positive sample / negative sample in the training sample set, inputting the assembly code in the positive sample / negative sample into the assembly encoder, and inputting the natural language text in the positive sample / negative sample into the text encoder;

[0034] Iteratively training the assembly encoder and the text encoder using the InfoNCE contrastive learning loss function to obtain a semantically aligned code representation learning model and a natural language representation learning model;

[0035] The code representation learning model can represent the input binary assembly code as a continuous vector, and the natural language representation learning model can represent the input natural language text as a continuous vector.

[0036] In a second aspect, the present application provides an application method of a representation learning model, comprising:

[0037] Inputting the binary code to be identified into the code representation learning model of the representation learning model, and determining a code representation vector corresponding to the binary code to be identified;

[0038] Generate corresponding natural language text according to each set behavior information / category information, where the behavior information / category information is multiple;

[0039] Inputting a plurality of natural language texts into a natural language representation learning model of the representation learning model, and determining a natural language representation vector corresponding to each natural language text;

[0040] Calculating vector similarity between the code representation vector and each natural language representation vector;

[0041] Determining a target vector among the natural language representation vectors that has the highest similarity to the code representation vector;

[0042] The behavior information / category information corresponding to the binary code to be identified is determined according to the behavior information / category information corresponding to the target vector.

[0043] In a third aspect, the present application provides an electronic device, comprising:

[0044] A sample generation module is used to extract multiple source codes from the open source repository and generate a training sample set based on the multiple source codes and the large language model, where each sample in the training sample set includes assembly code and natural language text;

[0045] The model training module is used to pre-train and generate an assembly encoder based on an assembly code dataset, and to pre-train and generate a text encoder based on a natural language dataset; the assembly encoder and the text encoder are aligned and trained based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model; and a representation learning model is constructed based on the semantically aligned code representation learning model and the natural language representation learning model.

[0046] In a fourth aspect, the present application provides another electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0047] The memory stores computer-executable instructions;

[0048] The processor executes the computer-executable instructions stored in the memory to implement the above method.

[0049] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the above-mentioned method when executed by a processor.

[0050] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0051] The training / application method, device and medium of the representation learning model provided in this application can extract multiple source codes from an open source repository, and generate a training sample set based on the multiple source codes and a large language model, where each sample in the training sample set includes assembly code and natural language text; pre-train an assembly encoder based on the assembly code dataset, and pre-train a text encoder based on the natural language dataset; align the assembly encoder and the text encoder based on the training sample set and a contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model; construct a representation learning model based on the semantically aligned code representation learning model and the natural language representation learning model. The method of this application extracts multiple source codes from an open source repository, processes the source codes to obtain a training sample set, and thus obtains a representation learning model based on the training sample set. Through such a setting, the generalization ability of the representation learning model is improved, so that the model can still show significant capabilities in few-sample and zero-sample learning scenarios, reduces dependence on samples, and improves the practicality of the model. Furthermore, each sample in the training sample set includes assembly code and natural language text, which can efficiently and large-scale generate binary code datasets and natural language datasets, avoiding the reliance on large amounts of labeled data during model training. The sample does not need to be labeled, thus reducing the cost and complexity of representation learning applications. Furthermore, based on the training sample set and the contrastive learning algorithm, the assembly encoder and the text encoder are aligned and trained. The paired data samples in the training sample set ensure that the representation learning model obtained by the final training can more comprehensively understand the semantic content of the binary code, thereby improving the accuracy and efficiency of representation learning. Aligning the assembly encoder and the text encoder using the contrastive learning method enables the representation learning model to more accurately learn and represent binary code, and also enhances the model's adaptability when processing new tasks and datasets, further improving the model's accuracy and generalization, ensuring that the model has sufficient generalization capabilities to handle completely unknown data types or tasks, and further improving the model's practicality and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] FIG1 is a flowchart of a training method for a representation learning model according to an embodiment of the present application;

[0054] FIG2 is a schematic diagram of a training sample set generation process according to an embodiment of the present application;

[0055] FIG3 is a schematic diagram of the training process of a representation learning model according to an embodiment of the present application;

[0056] FIG4 is a flow chart of a method for applying a representation learning model according to an embodiment of the present application;

[0057] FIG5 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

[0058] FIG6 is a schematic structural diagram of an electronic device according to another embodiment of the present application;

[0059] FIG7 is a schematic structural diagram of an electronic device according to another embodiment of the present application.

[0060] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0062] It should be noted that the training / application method, device and medium of the representation learning model of the present application can be used in the field of machine learning, and can also be used in any field other than the field of machine learning, such as the computer field, data processing field, etc. The application field of the training / application method, device and medium of the representation learning model of the present application is not limited.

[0063] The training / application method, device and medium of the representation learning model of the present application can be applied to any scenario related to binary code representation learning, such as function boundary detection, binary code search, binary code similarity detection, function prototype inference, malware classification, reverse engineering, encryption identification, protocol classification and algorithm classification, etc. The application scenarios of the training / application method, device and medium of the representation learning model of the present application are not limited.

[0064] First, let’s explain the terms involved in this application:

[0065] Assembly language is any low-level language, also known as a symbolic language, used for electronic computers, microprocessors, microcontrollers, or other programmable devices. In assembly language, mnemonics replace machine instruction opcodes, and address symbols or labels replace the addresses of instructions or operands. Different devices use different assembly language instruction sets, which are converted into machine instructions through the assembly process. A specific assembly language corresponds to a specific machine language instruction set and is not directly portable between different platforms. Different CPU architectures correspond to different assembly language versions.

[0066] Large language models (LLMs) are language models trained using large amounts of text data. As a proven approach, LLMs are characterized by extensive training datasets, a large number of model parameters and layers, and high computational complexity. Their value lies in their versatility and improved generalization, enabling them to handle a wide range of natural language tasks, such as text classification, question-answering, and conversation. Compared to traditional language models trained for specific domains, LLMs have a wider range of application scenarios.

[0067] Binary code representation learning is a method for representing binary code as continuous vectors in a vector space. Existing binary code representation learning methods are mainly categorized into three types: directly modeling raw bytes, using graphical models to establish control flow relationships between basic blocks, and representing function instructions as instruction sequences.

[0068] However, on the one hand, these three existing binary code representation learning methods still have shortcomings in representing the semantic information of binary codes and the control flow information of modeling programs, resulting in poor accuracy of representation learning.

[0069] (1) Although direct modeling methods based on raw bytes have advantages in computational efficiency, they perform poorly in capturing high-level semantic information and control flow graph (CFG) structure information of each instruction because they treat the function as a flat, continuous sequence of raw bytes and ignore instruction semantics and control flow information. In addition, these methods may encounter difficulties when processing binary codes with complex structures and semantics.

[0070] (2) Graphical model methods use graph neural networks (GNNs) to embed the control flow information of basic blocks into assembly code, thereby fusing control flow information into function representation for better function understanding. However, these methods often rely on manually selected features to represent control flow graph (CFG) blocks, which may result in insufficient semantic information being conveyed. In addition, since graphical model methods need to process graph structures, the computational complexity may increase significantly when processing large-scale binary codes, thus affecting the scalability of the methods.

[0071] (3) Sequence modeling methods represent function instructions as instruction sequences, which has the advantage of preserving instruction order. Although such methods perform well in capturing the semantic information of instructions, most methods have limitations in capturing control flow graph information. Some studies have proposed improved methods, such as jTrans, etc., which attempt to retain control flow information through improved models. However, during the tokenization and normalization process, these methods may lose call parameter information, disassembly string information, and external function call information. This limitation is a common problem of existing sequence modeling methods.

[0072] On the other hand, current methods often require large amounts of data for retraining when applied to new datasets or tasks. This reliance leads to poor performance when training samples are scarce. In particular, in few-shot learning scenarios, models must be able to quickly adapt to new tasks using very few examples, but existing methods often lack efficiency in this rapid adaptation capability. Furthermore, in zero-shot learning scenarios, where the model encounters a task it has never seen during training, current methods often perform even more limitedly. This is because these models are typically optimized specifically for their training data and lack sufficient generalization capabilities to handle completely unknown data types or tasks.

[0073] Furthermore, due to the complexity and diversity of binary code, as well as the rapid changes in programming practices, collecting and annotating large amounts of data for new tasks becomes extremely difficult and time-consuming. This increases the cost and complexity of applying these methods to real-world problems, especially in security-critical areas such as malware detection and vulnerability analysis, where the ability to quickly adapt to emerging threat patterns is crucial.

[0074] Based on this technical problem, the inventive concept of this application is: how to provide a representation learning method that can improve the generalization ability and accuracy of the representation learning model and reduce the dependence on samples and large amounts of labeled data.

[0075] Specifically, multiple source codes can be extracted from an open source repository, and a training sample set can be generated based on the multiple source codes and a large language model. Each sample in the training sample set includes assembly code and natural language text; an assembly encoder is pre-trained based on the assembly code dataset, and a text encoder is pre-trained based on the natural language dataset; the assembly encoder and the text encoder are aligned and trained based on the training sample set and a contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model; a representation learning model is constructed based on the semantically aligned code representation learning model and the natural language representation learning model. The method of the present application extracts multiple source codes from an open source repository and processes the source codes to obtain a training sample set, thereby training a representation learning model based on the training sample set. Through such a setting, the generalization ability of the representation learning model is improved, so that the model can still show significant capabilities in few-sample and zero-sample learning scenarios, reducing dependence on samples and improving the practicality of the model. Furthermore, each sample in the training sample set includes assembly code and natural language text, which can efficiently and large-scale generate binary code datasets and natural language datasets, avoiding dependence on a large amount of labeled data during model training, eliminating the need to label samples, and reducing the cost and complexity of representation learning applications. Furthermore, based on the training sample set and the contrastive learning algorithm, the assembly encoder and the text encoder are aligned and trained. The paired data samples in the training sample set ensure that the representation learning model finally trained can more comprehensively understand the semantic content of the binary code, thereby improving the accuracy and efficiency of the representation learning. The use of the contrastive learning method to align the assembly encoder and the text encoder enables the representation learning model to learn and represent the binary code more accurately, and also enhances the adaptability of the model when processing new tasks and data sets, further improving the accuracy and generalization of the model, ensuring that the model has sufficient generalization ability to handle completely unknown data types or tasks, and further improving the practicality and adaptability of the model.

[0076] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0077] Example 1

[0078] Figure 1 is a flow chart of a method for training a representation learning model according to an embodiment of the present application. This embodiment illustrates the method for training a representation learning model using an electronic device as the execution subject. As shown in Figure 1, the method for training a representation learning model may include the following steps:

[0079] S101: Extract multiple source codes from the open source repository and generate a training sample set based on the multiple source codes and the large language model.

[0080] In this embodiment, each sample in the training sample set may include assembly code and natural language text.

[0081] In this embodiment, the open source repository may be a large number of open source projects including source code. Those skilled in the art may flexibly set the specific open source repository as long as a large number of different source codes can be obtained from the open source repository.

[0082] In this embodiment, those skilled in the art can flexibly set the specific type of the large language model. For example, the large language model can be a ChatGPT model, a Wenyan Yixin model, an iFlytek Spark model, a ChatGLM model, an LLaMA model, etc., without any limitation here.

[0083] In this embodiment, the electronic device can be a terminal device such as a computer or a mobile phone, or a server device such as a physical server or a cloud server. The specific type of the electronic device can be flexibly set by those skilled in the art, and no restrictions are imposed here, as long as the electronic device can perform training and application of the representation learning model.

[0084] In one possible implementation, generating a training sample set based on multiple source codes and a large language model in step S101 may include:

[0085] S11: For each source code, generate different versions of binary assembly code according to the source code, and use the large language model to determine the natural language text corresponding to the source code.

[0086] S12: Generate a positive sample corresponding to the source code based on any binary assembly code corresponding to the source code and a natural language text corresponding to the source code.

[0087] S13: Generate negative samples corresponding to the source code based on any binary assembly code corresponding to the source code and any natural language text not corresponding to the source code, where there are multiple negative samples and the number of negative samples is greater than the number of positive samples.

[0088] S14: Generate a positive sample set based on the positive sample corresponding to each source code, and generate a negative sample set based on the negative sample corresponding to each source code.

[0089] S15: Generate a training sample set based on the positive sample set and the negative sample set.

[0090] In this embodiment, different CPU architectures correspond to different assembly versions. The specific version of the binary assembly code can be flexibly set by those skilled in the art according to actual conditions and is not limited here.

[0091] In this implementation, binary assembly code and natural language text generated from the same source code constitute positive samples (semantically identical), while binary assembly code and natural language text generated from different source code constitute negative samples (semantically different). The number of negative samples can be set to be much larger than the number of positive samples, thereby improving the accuracy of the model's semantic representation.

[0092] In this implementation, a large language model and a large amount of source code can be used to generate positive samples and a large number of negative samples. Positive samples include semantically identical assembly code and natural language text, while negative samples include semantically different assembly code and natural language text. This setup allows for a very large pool of negative samples, from which multiple pairs of negative samples can be randomly selected. This ensures that the model is exposed to a large number of negative samples during learning, allowing for better semantic alignment between the source code and the self-verifying oracle description, improving model performance.

[0093] In one possible implementation, the use of the large language model in step S11 to determine the natural language text corresponding to the source code may include: using the pre-trained large language model as the target model, using the shadow model strategy to train a shadow application model corresponding to the target model; and determining the natural language text corresponding to the source code based on the shadow application model.

[0094] In this embodiment, when training using the shadow model strategy, task-specific data can be used for training, which can be natural language description data obtained from requests from large models (such as GPT-3.5, etc.).

[0095] For example, Figure 2 is a schematic diagram of the training sample set generation process in one embodiment of the present application. As shown in Figure 2, after extracting multiple source codes from an open source repository, the source codes can be compiled into different versions of binary assembly code. Furthermore, the natural language text corresponding to the source code can be determined based on the shadow model of the large language model. Based on the sources and correspondence between the assembly code and the natural language text, positive and negative sample sets can be generated, thereby generating a training sample set.

[0096] In this implementation, directly using a large language model to determine the natural language text corresponding to the source code is expensive. Therefore, a shadow model strategy can be used to train a shadow application model corresponding to the large language model. The lower-cost shadow application model can then be used to determine the natural language text corresponding to the source code. The shadow model's primary goal is to mimic the performance of a model that has been highly effective at interpreting code, but with greater resource efficiency. This reduces model training costs while ensuring effective training.

[0097] S102: Pre-training an assembly encoder based on an assembly code dataset, and pre-training a text encoder based on a natural language dataset.

[0098] In this embodiment, the assembly code dataset can be any training set including assembly code, and those skilled in the art can flexibly set it. The assembly code dataset can be a dataset composed of different versions of binary assembly code generated in the above step S11, or a dataset composed of assembly code obtained by other means (open source or non-open source dataset), and no restrictions are made here.

[0099] In this embodiment, the assembly encoder is pre-trained on a large number of assembly code datasets, so that the model can effectively learn and extract key features in various assembly language environments, significantly improving the model's ability to understand and adapt to binary codes.

[0100] Similarly, the natural language dataset can be any training set that includes natural language text. Those skilled in the art can flexibly set it. The natural language dataset can be a dataset composed of natural language text generated in the above step S11, or a dataset composed of natural language text obtained by other means (open source or non-open source dataset). No restrictions are imposed here.

[0101] In this embodiment, the specific type of the text encoder can be flexibly set by those skilled in the art. Preferably, in order to improve the model performance, the text encoder can be an existing text encoder with better text similarity detection and more accurate semantic judgment, and no restrictions are made here.

[0102] In one possible implementation, pre-training and generating an assembly encoder based on the assembly code dataset in step S102 may include:

[0103] S21: For each assembly code data in the assembly code data set, relocate the address of the assembly code data so that the assembly code data maintains a relative address relationship when processing a jump instruction.

[0104] S22: Using a word segmentation algorithm to segment the relocated assembly code data into independent tokens to obtain code samples corresponding to the assembly code data.

[0105] S23: Generate a code sample set according to the code sample corresponding to each assembly code data, and generate an assembly encoder through pre-training based on the code sample set.

[0106] In this embodiment, the word segmentation algorithm may be the WordPiece word segmentation algorithm or other word segmentation algorithms, and no limitation is made here.

[0107] In this embodiment, when pre-training and generating an assembly encoder, the address of the assembly code in the assembly code dataset can be relocated first, and the relative address relationship can be maintained when processing jump instructions. Through such a setting, the control flow information in the assembly code is retained, and the relative distance between the jump instruction and its target is maintained during the relocation process, so that the model can understand the control flow changes caused by the jump instruction, thereby improving the accuracy of the assembly encoder code representation learning. Furthermore, after the address relocation, the assembly code can be segmented into independent tokens, that is, the WordPiece algorithm is used to train the word segmenter on the entire assembly code dataset, which can losslessly encode the assembly code without the need for standardization processing, thereby retaining key information, such as call parameters and external function names, etc., further improving the accuracy of the assembly encoder code representation learning.

[0108] In one possible implementation, generating an assembly encoder based on the code sample set pre-training in step S23 may include:

[0109] S31: Generate the initial encoder based on the Transformer model architecture.

[0110] S32: Determine the instruction embedding corresponding to each instruction in the initial encoder.

[0111] S33: Embed the instructions corresponding to each instruction and integrate them into the initial encoder to generate an intermediate encoder.

[0112] S34: Pre-training the intermediate encoder according to each code sample in the code sample set, and sharing the tag embedding parameters of the jump symbol and its corresponding instruction embedding during the model pre-training process to obtain an assembly encoder.

[0113] In this embodiment, the specific structure of the assembly encoder can refer to the Transformer model that has performed well in binary code analysis, such as the RoBERTa model, which contains approximately 110 million parameters. Of course, other models based on the Transformer model architecture can also be used. Those skilled in the art can flexibly configure it, and no limitations are imposed here.

[0114] In this embodiment, by integrating the instruction embedding corresponding to each instruction into the initial encoder, the boundaries of the assembly instructions can be clearly demarcated, so that the model can effectively identify and process individual instructions. By assigning a unique embedding to each instruction, the model learns to distinguish different instructions, providing a basis for subsequent model training. Furthermore, by sharing the tag embedding parameters of the jump symbol and its corresponding instruction embedding, the model's understanding of the jump relationship within the assembly code is enhanced. For example, specific tag embeddings share parameters with instruction embeddings, enabling the model to capture the complex connections between the various components of the assembly code. Through shared parameters, the model learns to associate symbolic tags with their target tags, thereby better understanding the control flow within the code, further improving the accuracy of the assembly encoder code representation learning, and improving model performance.

[0115] S103: Based on the training sample set and the contrastive learning algorithm, the assembly encoder and the text encoder are aligned and trained to obtain a semantically aligned code representation learning model and a natural language representation learning model.

[0116] In one possible implementation, step S103 performs alignment training on the assembly encoder and the text encoder based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model, which may include:

[0117] S41: For each positive sample / negative sample in the training sample set, the assembly code in the positive sample / negative sample is input into the assembly encoder, and the natural language text in the positive sample / negative sample is input into the text encoder.

[0118] S42: Using the InfoNCE contrastive learning loss function, the assembly encoder and text encoder are iteratively trained to obtain semantically aligned code representation learning models and natural language representation learning models.

[0119] Among them, the code representation learning model can represent the input binary assembly code as a continuous vector, and the natural language representation learning model can represent the input natural language text as a continuous vector.

[0120] In this embodiment, the InfoNCE contrastive learning loss function is used to iteratively train the assembly encoder and the text encoder, which can enhance the model's ability to distinguish between positive and negative samples, maximize the mutual information between positive sample pairs and negative samples, and enable the model to more accurately learn and represent binary codes. This not only improves the accuracy of the model, but also enhances its adaptability when processing new tasks and data sets, further improves the accuracy and generalization of the model, ensures that the model has sufficient generalization ability to handle completely unknown data types or tasks, and further improves the practicality and adaptability of the model.

[0121] S104: Construct a representation learning model based on the semantically aligned code representation learning model and the natural language representation learning model.

[0122] Exemplarily, Figure 3 is a schematic diagram of the training process of the representation learning model of an embodiment of the present application. As shown in Figure 3, the assembly code in the training sample set (positive samples / negative samples) can be input into the assembly encoder, and the natural language text can be input into the text encoder, and iterative training is performed using the InfoNCE comparative learning loss function to obtain a semantically aligned representation learning model.

[0123] In this embodiment, following steps S101-S105 above, a representation learning model is constructed that learns binary code representations by aligning binary code with natural language text. This method, which includes an efficient data engine and a powerful assembly encoder, successfully bridges the gap between assembly code and natural language representations. The resulting model demonstrates significant zero-shot and few-shot learning capabilities across a variety of tasks, including binary code similarity detection, encryption recognition, protocol classification, and algorithm classification.

[0124] It will be readily understood by those skilled in the art that the above is only a preferred embodiment of the present application, and that several improvements and modifications may be made without departing from the principles of the present application. For example, although we have used a specific compiler and optimization options in this embodiment, other compilers and optimization options may be used. Similarly, although we have used specific pre-training tasks and training tasks here, other types of pre-training tasks and training tasks may be used. Furthermore, although we have used a specific network structure here, other types of network structures may be used.

[0125] In this embodiment, multiple source codes can be extracted from an open source repository and processed to generate a training sample set. This training sample set is then used to train a representation learning model. This setup improves the generalization capability of the representation learning model, enabling the model to demonstrate significant capabilities in few-shot and zero-shot learning scenarios, reducing reliance on samples and improving the model's practicality. Furthermore, each sample in the training sample set includes both assembly code and natural language text, enabling efficient and large-scale generation of binary code and natural language datasets. This avoids the reliance on large amounts of labeled data during model training, eliminates the need for sample labeling, and reduces the cost and complexity of representation learning applications. Furthermore, based on the training sample set and the contrastive learning algorithm, the assembly encoder and the text encoder are aligned and trained. The paired data samples in the training sample set ensure that the representation learning model finally trained can more comprehensively understand the semantic content of the binary code, thereby improving the accuracy and efficiency of the representation learning. The use of the contrastive learning method to align the assembly encoder and the text encoder enables the representation learning model to learn and represent the binary code more accurately, and also enhances the adaptability of the model when processing new tasks and data sets, further improving the accuracy and generalization of the model, ensuring that the model has sufficient generalization ability to handle completely unknown data types or tasks, and further improving the practicality and adaptability of the model.

[0126] Example 2

[0127] Figure 4 is a flow chart of an application method of a representation learning model according to an embodiment of the present application. This embodiment illustrates the application method of the representation learning model using an electronic device as the execution subject. As shown in Figure 4, the application method of the representation learning model may include the following steps:

[0128] S201: Inputting the binary code to be recognized into the code representation learning model of the representation learning model to determine the code representation vector corresponding to the binary code to be recognized.

[0129] In this embodiment, the binary code to be identified may be a code of unknown behavior / category. By using the representation learning model, the behavior / category corresponding to the binary code to be identified may be determined for subsequent operations.

[0130] S202: Generate corresponding natural language text according to each set behavior information / category information, where there are multiple behavior information / category information.

[0131] In this embodiment, the set behavior information / category information can be several types of behavior information / category information that are close to the binary code to be identified or detected by the staff, such as various common malicious behaviors such as screenshots, file uploads, and keylogging. The specific behavior information / category information can be flexibly set by those skilled in the art and is not limited here.

[0132] S203: Inputting a plurality of natural language texts into a natural language representation learning model of a representation learning model, and determining a natural language representation vector corresponding to each natural language text.

[0133] S204: Calculate the vector similarity between the code representation vector and each natural language representation vector.

[0134] S205: Determine a target vector in the natural language representation vector that has the highest similarity to the code representation vector.

[0135] In this embodiment, the higher the vector similarity between the code representation vector and the natural language representation vector, the closer their semantics are.

[0136] S206: Determine the behavior information / category information corresponding to the binary code to be identified based on the behavior information / category information corresponding to the target vector.

[0137] It should be noted that since the representation learning model trained in Example 1 performs well in a variety of downstream tasks, it is not only limited to the application scenarios of this embodiment, but can also be widely used in multiple fields of binary code analysis and processing. This embodiment is only an example of an application of the representation learning model.

[0138] In this embodiment, when determining the behavior / category corresponding to the binary code to be identified, the binary code to be identified can first be input into the code representation learning model to obtain a code representation vector, and then the natural language text corresponding to the set behavior information / category information can be input into the natural language representation learning model to obtain a natural language representation vector. Because the code representation learning model and the natural language representation learning model are semantically aligned using a contrastive learning method, they can not only learn and represent binary codes more accurately, but also enhance the model's adaptability when processing new tasks and data sets. Therefore, after obtaining the representation vector, the behavior information / category information corresponding to the binary code to be identified can be simply and accurately determined based on the behavior information / category information corresponding to the natural language representation vector with the highest similarity to the code representation vector.

[0139] The following is an explanation of the training and application methods of the representation learning model of the present application using a specific embodiment.

[0140] Example 3

[0141] In a specific embodiment, a software company wants to determine the behavior corresponding to a malicious assembly code. First, it needs to vectorize the malicious assembly code. The specific process is as follows:

[0142] In the first step, the staff input a large number of open source projects including source code into the computer and input model building instructions. The computer extracts multiple source codes from the open source warehouse and generates a training sample set based on multiple source codes and a large language model. Each sample in the training sample set includes assembly code and natural language text.

[0143] In the second step, the computer pre-trains and generates an assembly encoder based on the set assembly code dataset, and pre-trains and generates a text encoder based on the set natural language dataset.

[0144] In the third step, the computer aligns the assembly encoder and the text encoder based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model.

[0145] In the fourth step, the computer constructs a representation learning model based on the semantically aligned code representation learning model and the natural language representation learning model.

[0146] In the fifth step, the staff inputs the malicious assembly code and the behavior information to be detected into the computer's representation learning model and starts the recognition task. The computer inputs the malicious assembly code into the code representation learning model to determine the corresponding code representation vector.

[0147] In the sixth step, the computer generates corresponding natural language text based on each behavioral information to be detected, inputs multiple natural language texts into the natural language representation learning model, and determines the natural language representation vector corresponding to each natural language text.

[0148] In the seventh step, the computer calculates the vector similarity between the code representation vector and each natural language representation vector, and determines the target vector in the natural language representation vector that has the highest similarity with the code representation vector.

[0149] In the eighth step, the computer determines that the behavior information corresponding to the malicious assembly code is a backdoor based on the behavior information corresponding to the target vector.

[0150] FIG5 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. As shown in FIG5 , the electronic device includes: a sample generation module 51 for extracting multiple source codes from an open source repository, and generating a training sample set based on the multiple source codes and a large language model, wherein each sample in the training sample set includes assembly code and natural language text; a model training module 52 for pre-training and generating an assembly encoder based on an assembly code dataset, and pre-training and generating a text encoder based on a natural language dataset; performing alignment training on the assembly encoder and the text encoder based on the training sample set and a contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model; and constructing a representation learning model based on the semantically aligned code representation learning model and the natural language representation learning model. In one embodiment, the description of the specific implementation functions of the electronic device can be found in steps S101-S104 in Example 1, which will not be repeated here.

[0151] FIG6 is a schematic diagram of the structure of an electronic device according to another embodiment of the present application. As shown in FIG6 , the electronic device includes: a vector generation module 61 for inputting a binary code to be identified into a code representation learning model of a representation learning model to determine a code representation vector corresponding to the binary code to be identified; generating a corresponding natural language text according to each set behavior information / category information, wherein the behavior information / category information is multiple; inputting multiple natural language texts into a natural language representation learning model of the representation learning model to determine a natural language representation vector corresponding to each natural language text; a matching detection module 62 for calculating the vector similarity between the code representation vector and each natural language representation vector; determining a target vector in the natural language representation vector that has the highest similarity to the code representation vector; and determining the behavior information / category information corresponding to the binary code to be identified according to the behavior information / category information corresponding to the target vector. In one embodiment, the description of the specific functions implemented by the electronic device can refer to steps S201-S206 in Example 2 and will not be repeated here.

[0152] Figure 7 is a structural diagram of an electronic device of another embodiment of the present application. As shown in Figure 7, the electronic device includes: a processor 101, and a memory 102 communicatively connected to the processor 101; the memory 102 stores computer execution instructions; the processor 101 executes the computer execution instructions stored in the memory 102 to implement the steps of the training / application method representing the learning model in the above-mentioned method embodiments.

[0153] In the electronic device described above, the memory 102 and the processor 101 are electrically connected, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines, such as a bus connection. The memory 102 stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the memory 102 in the form of software or firmware. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102.

[0154] The memory 102 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 102 is used to store programs, and the processor 101 executes the programs after receiving execution instructions. Furthermore, the software programs and modules in the memory 102 may also include an operating system, which may include various software components and / or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0155] The processor 101 can be an integrated circuit chip with signal processing capabilities. The processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0156] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the steps of each method embodiment of the present application.

[0157] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of each method embodiment of the present application when executed by a processor.

[0158] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0159] It should be further noted that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0160] It should be understood that the above-described device embodiments are merely illustrative, and the device of the present application may also be implemented in other ways. For example, the division of units / modules in the above-described embodiments is merely a logical functional division, and actual implementations may employ other division methods. For example, multiple units, modules, or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0161] In addition, unless otherwise specified, the functional units / modules in the various embodiments of the present application may be integrated into a single unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The aforementioned integrated units / modules may be implemented in the form of hardware or software program modules.

[0162] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined in any way. To keep the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the appended claims.

[0164] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A training method for a representation learning model, characterized in that: include: Extracting multiple source codes from an open source repository and generating a training sample set based on the multiple source codes and a large language model, wherein each sample in the training sample set includes assembly code and natural language text; Pre-train an assembly encoder based on an assembly code dataset, and pre-train a text encoder based on a natural language dataset. Performing alignment training on the assembly encoder and the text encoder based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model; Constructing a representation learning model based on the semantically aligned code representation learning model and the natural language representation learning model; The pre-training and generating of the assembly encoder according to the assembly code dataset specifically includes: For each assembly code data in the assembly code data set, relocating the addresses of the assembly code data so that the assembly code data maintains a relative address relationship when processing a jump instruction; Using a word segmentation algorithm to segment the relocated assembly code data into independent tokens to obtain code samples corresponding to the assembly code data; Generate a code sample set according to the code sample corresponding to each assembly code data; Generate the initial encoder based on the Transformer model architecture; Determining an instruction embedding corresponding to each instruction in the initial encoder; Embedding the instructions corresponding to each instruction into the initial encoder to generate an intermediate encoder; The intermediate encoder is pre-trained according to each code sample in the code sample set, and the tag embedding parameters of the jump symbol and the corresponding instruction embedding are shared during the model pre-training process to obtain an assembly encoder.

2. The method according to claim 1, characterized in that Generating a training sample set based on multiple source codes and a large language model specifically includes: For each source code, generating different versions of binary assembly code according to the source code, and determining the natural language text corresponding to the source code using a large language model; Generate a positive sample corresponding to the source code according to any binary assembly code corresponding to the source code and a natural language text corresponding to the source code; Generate negative samples corresponding to the source code according to any binary assembly code corresponding to the source code and any natural language text not corresponding to the source code, wherein there are multiple negative samples and the number of negative samples is greater than the number of positive samples; Generate a positive sample set based on the positive sample corresponding to each source code, and generate a negative sample set based on the negative sample corresponding to each source code; A training sample set is generated according to the positive sample set and the negative sample set.

3. The method according to claim 2, characterized in that The determining of the natural language text corresponding to the source code by using the large language model specifically includes: Taking the pre-trained large language model as the target model, the shadow model strategy is used to train the shadow application model corresponding to the target model; Determine a natural language text corresponding to the source code according to the shadow application model.

4. The method according to claim 1, wherein The alignment training of the assembly encoder and the text encoder based on the training sample set and the contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model specifically includes: For each positive sample / negative sample in the training sample set, inputting the assembly code in the positive sample / negative sample into the assembly encoder, and inputting the natural language text in the positive sample / negative sample into the text encoder; Iteratively training the assembly encoder and the text encoder using the InfoNCE contrastive learning loss function to obtain a semantically aligned code representation learning model and a natural language representation learning model; The code representation learning model can represent the input binary assembly code as a continuous vector, and the natural language representation learning model can represent the input natural language text as a continuous vector.

5. A method for applying a representation learning model, wherein the representation learning model is trained using the training method for a representation learning model according to any one of claims 1 to 4, characterized in that: include: Inputting the binary code to be identified into the code representation learning model of the representation learning model, and determining a code representation vector corresponding to the binary code to be identified; Generate corresponding natural language text according to each set behavior information / category information, where the behavior information / category information is multiple; Inputting a plurality of natural language texts into a natural language representation learning model of the representation learning model, and determining a natural language representation vector corresponding to each natural language text; Calculating vector similarity between the code representation vector and each natural language representation vector; Determining a target vector among the natural language representation vectors that has the highest similarity to the code representation vector; The behavior information / category information corresponding to the binary code to be identified is determined according to the behavior information / category information corresponding to the target vector.

6. An electronic device comprising: A sample generation module is used to extract multiple source codes from the open source repository and generate a training sample set based on the multiple source codes and the large language model, where each sample in the training sample set includes assembly code and natural language text; A model training module is configured to pre-train an assembly encoder based on an assembly code dataset and pre-train a text encoder based on a natural language dataset; align the assembly encoder and the text encoder based on the training sample set and a contrastive learning algorithm to obtain a semantically aligned code representation learning model and a natural language representation learning model; and construct a representation learning model based on the semantically aligned code representation learning model and the natural language representation learning model. The model training module is specifically configured to relocate the address of each assembly code data in the assembly code data set so that the assembly code data maintains a relative address relationship when processing a jump instruction; and use a word segmentation algorithm to segment the relocated assembly code data into independent tokens to obtain code samples corresponding to the assembly code data; Generate a code sample set according to the code sample corresponding to each assembly code data; Generating an initial encoder according to the Transformer model architecture; determining an instruction embedding corresponding to each instruction in the initial encoder; integrating the instruction embedding corresponding to each instruction into the initial encoder to generate an intermediate encoder; According to each code sample in the code sample set, the intermediate encoder is pre-trained, and the tag embedding parameters of the jump symbol and its corresponding instruction embedding are shared during the model pre-training process to obtain an assembly encoder.

7. An electronic device, characterized in that: comprising a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 5 when executed by a processor.

Images