Cross-CPU (central processing unit) architecture binary file semantic segmentation and entity identification method for credential and credential adaptation
Automatic analysis of ELF files through AI model solves problems such as high requirements for binary file parsing, difficulty in handling irregular files, and difficulty in adapting to new CPU structures in the existing technology, and efficient and automated semantic segmentation and entity recognition of binary files are achieved, promoting the promotion of efficient and automated binary file semantic segmentation and entity recognition. The development of the information innovation ecosystem.
Patent Information
- Application Number
- CN202411561509.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art has multiple limitations when processing binary executable files, including high requirements for ELF or PE file structure, difficult to deal with files with irregular formats or obfuscated formats, difficulty in adapting to new CPU structures or instruction set modifications, and inability to automatically resolve them. , high threshold and high development and maintenance costs.
Using AI model to realize the automatic analysis of ELF files, through data acquisition and labeling, data loading and preprocessing, the construction of binary file segmentation classification model and Bin-Parser model, combined with GloVe vector embedding, the semantic segmentation of binary files across CPU architecture and Entity identification.
It greatly improves work efficiency, reduces labor costs, can quickly respond to and process a large number of documents, adapt to rapidly changing industry needs, and is conducive to the development of the information and innovation ecosystem.
Smart Images
Figure CN119473386A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of program compilation, and in particular to a method for semantic segmentation and entity recognition of binary files across CPU architectures for information technology adaptation. Background Art
[0002] The existing binary executable file parsing method is to parse the binary file format logically according to the ELF file structure or PE file structure to extract the instructions, data and metadata. Parse the file header, segment table and symbol table, as well as instructions and other information. This step requires a deep understanding of the ELF or PE file structure, and this type of rule-based parsing method has many limitations, especially for trust creation and new instruction architectures.
[0003] The problems existing in the existing technology are:
[0004] (1) Rule-based methods have high requirements on people, are difficult to implement, and are inefficient.
[0005] (2) It can only process ELF or PE files that comply with the standards, and cannot process files that are not in the standard format or are obfuscated.
[0006] (3) It is difficult to expand and modify the tools for new CPU structures or instruction sets.
[0007] (4) It is difficult to adapt to new file formats or variants, and the parsing rules need to be manually updated.
[0008] (5) It requires relatively high software technical skills and has a high threshold, which ordinary people do not have, hindering the development of the ecosystem and increasing the cost of development and maintenance. Summary of the invention
[0009] The purpose of this invention is to overcome the shortcomings of the prior art, propose a cross-CPU architecture binary file semantic segmentation and entity recognition method for trusted innovation adaptation, and realize the automation of ELF file parsing through an AI model, which can greatly improve the work efficiency of the industry and reduce labor costs; the AI model can quickly respond to and process a large number of files, adapt to rapidly changing industry needs, and is conducive to the development of the trusted innovation ecosystem.
[0010] The present invention solves the technical problem by adopting the following technical solutions:
[0011] The cross-CPU architecture binary file semantic segmentation and entity recognition method for trusted innovation adaptation includes the following steps:
[0012] Step 1: Data collection and annotation;
[0013] Step 2: Load and preprocess the data;
[0014] Step 3: construct a binary file segment classification model, and classify the loaded data using the binary file segment classification model;
[0015] Step 4: Input the classified data into the segment loading module for processing;
[0016] Step 5: Input the data processed by the segment loading module into the Bin-Parser model and GloVe vector embedding respectively. The output of the Bin-Parser model is combined with the GloVe model training corpus generated by GloVe vector embedding to output semantic segmentation and annotated text.
[0017] Moreover, the specific implementation method of step 1 is: collect a variety of ARM and x86, RISC-V binary files, including application programs, libraries and operating system binary files; these files are used as input files, and the pyelftools tool is used to automatically annotate the key semantic units and entities in the ELF file, including functions, variables, segments, opcodes, operands, symbol tables and character tables.
[0018] Moreover, the specific implementation method of step 2 is: use the pyelftools tool to parse the collected binary files, and extract the required part of the content according to the structure of the binary files:
[0019] Step 2.1, parse ELF file: use pyelftools to parse ELF file and extract detailed information of each part;
[0020] Step 2.2, Generate annotation data: Generate detailed annotation data for each part, including the label of each byte;
[0021] Step 2.3, Save the annotation data: Save the annotation data in JSON format for subsequent use.
[0022] Moreover, the binary file segment classification model constructed in the step 3 includes: a 1-dimensional convolution layer, a pooling layer, an LSTM layer, a fully connected layer and a final fully connected layer, wherein the input passes through the convolution layer and the ReLU activation function, and then through the pooling layer, the input passes through the LSTM layer, the output of the last time step is taken, and the output of the LSTM passes through the fully connected layer and the ReLU activation function; the global information is combined with the local features, and the classification result is output through the final fully connected layer, and the classification result ELF file constitutes the classification code.
[0023] Furthermore, the step 4 comprises the following steps:
[0024] Step 4.1, binary file segment classification model result conversion: get the processing result from the binary file segment classification model, and calculate the starting position and offset size of each segment according to the processing result;
[0025] Step 4.2, binary data reading: read the binary data from the starting position and offset obtained above according to the binary format;
[0026] Step 4.3, data distribution: Distribute the read data to the GloVe vector embedding and Bin-Parser models respectively.
[0027] Moreover, the processing method of GloVe vector embedding in step 5 is: using the libelf library to read and parse the ELF file, perform actions similar to word segmentation, and organize it into separate function calls and code segments for training to build a corpus; using the instruction set specification and the ABI specification to establish a mapping between binary and instruction sets, and construct a co-occurrence matrix of Glove; using the collected expected information to train the Glove model, the Glove model saves the word embedding vector that reflects the semantics of the binary file; saving the Glove model as a file.
[0028] The advantages and positive effects of the present invention are:
[0029] The present invention first uses a binary file segment classification model to identify and divide large binary files into regions, solves the processing of large file long sequences, and then uses similar Transformer technology, for instruction sets, architecture specifications, CPU level ABI specifications and underlying ABI interface specifications (libc, glibc, libc++, libstdc++, etc.), ELF file structure specifications and other information through large data volume Model training, train an Encoder model, Encoder is used to read the ELF file information and auxiliary information of the source CPU architecture, and the binary semantic information collected by the Encoder model is used to perform semantic annotation and entity recognition of binary files. The ELF and PE file parsing methods based on AI of the present invention can promote the development of new tools and platforms, and improve the overall technical level of the industry; At the same time, the flexibility of the AI model enables it to be applied to multiple fields, such as network security, reverse engineering, software analysis, etc., to promote cross-domain technology integration and innovation, which is conducive to the development of the letter creation ecology; The present invention realizes the automation of ELF file parsing through the AI model, which can greatly improve the work efficiency of the industry and reduce labor costs; The AI model can quickly respond to and process a large number of files, adapt to the rapidly changing industry needs, and is conducive to the development of the letter creation ecology. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a flow chart of the present invention;
[0031] Figure 2 The binary file segment classification model structure of the present invention;
[0032] Figure 3 This is a flow chart of the segment loading module processing of the present invention;
[0033] Figure 4 GloVe vector embedding process flow chart of the present invention;
[0034] Figure 5 It is the structure diagram of Bin-Parser model of the present invention;
[0035] Figure 6 It is a processing flow chart of the Bin-Parser model of the present invention. DETAILED DESCRIPTION
[0036] The present invention is further described in detail below with reference to the accompanying drawings.
[0037] Semantic segmentation and entity recognition methods for binary files across CPU architectures for trusted innovation, such as Figure 1 As shown, the following steps are included:
[0038] Step 1: Data collection and labeling.
[0039] Collect a variety of ARM and x86, RISC-V binary files, including applications, libraries and operating system binaries. These files are used as input files, and the pyelftools tool is used to automatically annotate the key semantic units and entities in the ELF file, such as functions, variables, segments, opcodes, operands, symbol tables, character tables, etc.
[0040] Step 2: Load and preprocess the data. Use the pyelftools tool to parse the collected binary files and extract the required content according to the structure of the binary files.
[0041] In the training phase of the binary file segment classification model, pyelftools is needed to read the start length and offset of each segment, as well as the global information of the file, such as the file length, the number of segments, and the number of sections. The output is a JSON file as the annotation data. One annotation data corresponds to one ELF file. The read data is binary byte information. In actual training, the long file is split into blocks of 8K size suitable for the model input length, and the sliding window technology is used to retain the context information to ensure semantic integrity.
[0042] The main structure of the ELF file:
[0043] ELF header: The beginning of the file, containing basic information such as file type, architecture, entry point address, etc.
[0044] Program header table: describes the information of program segments (such as code segments, data segments), which is used for program loading.
[0045] Section header table: describes the information of the section (such as .text, .data, .bss, etc.), which is used for linking and debugging.
[0046] Actual segments and section data: Contains actual content such as code, data, symbol table, string table, etc.
[0047] The specific implementation method of this step is:
[0048] Parsing ELF files: Use pyelftools to parse ELF files and extract detailed information of each part.
[0049] Generate annotation data: Generate detailed annotation data for each part, including labels for each byte.
[0050] Save annotation data: Save the annotation data in JSON format for later use.
[0051] Step 3: Build a binary file segment classification model, and classify the loaded data using the binary file segment classification model.
[0052] Because ELF and PE have their own strict structures, and the content and length of different structures are different, in order to be able to read and analyze the entire binary file at one time, the model adopts a hierarchical structure model. First, a small model is trained to analyze the overall structure (binary file segment classification model), and then each part is trained and inferred using the transformer model.
[0053] like Figure 2 As shown in the figure, the binary file segment classification model includes a 1D convolution layer (Conv1d), a pooling layer (MaxPool1d), an LSTM layer, a fully connected layer (Linear), and finally a fully connected layer (Linear). The input passes through the convolution layer and the ReLU activation function, and then through the pooling layer. The input passes through the LSTM layer and the output of the last time step is taken. The output of the LSTM passes through the fully connected layer and the ReLU activation function.
[0054] Step 3.1: Combine global information with local features, pass through the final fully connected layer, output the classification result, and the classification code composed of the classification result ELF file.
[0055] Step 3.2: The model input is a binary file segment segmented according to the sliding window, with a segment length of 8K. The pre-made training data program reads the data through pyelftools and automatically annotates the data. The model loads the training data for training, parameter adjustment, and optimization, and generates a binary file segment classification model.
[0056] Step 4: Input the classified data into the segment loading module for processing.
[0057] like Figure 3 As shown, this step includes the following steps:
[0058] Step 4.1, binary file segment classification model result conversion: get the processing result from the binary file segment classification model, and calculate the starting position and offset size of each segment according to the processing result;
[0059] Step 4.2, binary data reading: read the binary data from the starting position and offset obtained above according to the binary format;
[0060] Step 4.3, data distribution: Distribute the read data to the GloVe vector embedding and Bin-Parser models respectively.
[0061] Step 5: Input the data processed by the segment loading module into the Bin-Parser model and GloVe vector embedding respectively. The output of the Bin-Parser model is combined with the GloVe model training corpus generated by GloVe vector embedding to output semantic segmentation and annotated text.
[0062] like Figure 4 As shown, the processing method of GloVe vector embedding is:
[0063] Data processing: Use the libelf library to read and parse the ELF file, perform actions similar to word segmentation (read different sections of ELF separately, extract opcodes and operands, as well as other binary file related information such as symbol tables), and organize them into separate function calls and code segments that can be used for training to build a corpus.
[0064] Data encoding: Use the instruction set specification and ABI specification to establish a mapping between binary and instruction set, and construct the co-occurrence matrix of Glove.
[0065] Semantic training: The Glove model is trained using the collected corpus information. The Glove model saves word embedding vectors that reflect the semantics of binary files.
[0066] Data feature vector generation: Save the Glove model as a file.
[0067] Bin-Parser Model:
[0068] The binary file semantic segmentation and entity recognition model is based on the Transform-Xl model and combines the convolutional attention mechanism to increase the capture of local features. During the training process, the training method is similar to the Chinese word segmentation and entity recognition training method, and the goal is achieved through parameter adjustment and optimization. This model only relies on the Encoder part, and the Decoder part does not need to be trained.
[0069] like Figure 5 As shown, the characteristics of the Bin-Parser model are:
[0070] Convolutional Self-Attention Module (CSA):
[0071] Convolutional layer: used to extract local features.
[0072] Self-attention layer: used to capture global dependencies.
[0073] Fusion mechanism: combining convolutional features and self-attention features.
[0074] Transformer-XL architecture:
[0075] Memory module: used for long-term dependency modeling.
[0076] Relative position encoding: Enhancing temporal consistency.
[0077] like Figure 6 As shown in the figure, the processing process of the Bin-Parser model is:
[0078] Input Embeddings:
[0079] Input: Byte sequence feature vector of binary file.
[0080] Output: Convert the input data into an embedded representation that the model can process.
[0081] Positional Encoding:
[0082] Input: Sequence from input embedding.
[0083] Output: Add position information to each input embedding to help the model understand the positional relationship of each element in the sequence.
[0084] Convolutional Attention Layer:
[0085] Input: Positionally encoded embedding vector.
[0086] Output: Attention-weighted features extracted from local features.
[0087] Processing: Use convolution operations to capture local features in the sequence, and combine attention mechanisms to weight important features. This step is a combination of traditional convolutional layers and attention mechanisms, which is a process of feature extraction and attention weight allocation for local areas.
[0088] Multi-Head Attention:
[0089] Input: Output of the convolutional attention layer.
[0090] Output: Feature representation that incorporates global context information.
[0091] Processing: The self-attention mechanism with multiple “heads” processes the input sequence in parallel, and each head learns different representation subspace information. In this way, the model is able to capture different aspects of the sequence in different representation subspaces.
[0092] Feed-Forward Network:
[0093] Input: The output of the multi-head self-attention layer, usually a high-dimensional feature representation.
[0094] Output: Feature representation after two linear transformations (with activation functions). This output will be sent back to the layer for normalization and residual connection.
[0095] Processing: Feedforward networks usually contain two linear transformations, the first one followed by an activation function (ReLU), and the second one without an activation function. This process can be expressed as: FFN(x) = max(0,xW1+b1)W2+b2, where W1, W2 are weight matrices, b1, b2 are bias vectors
[0096] Layer Normalization:
[0097] Input: Output of the feed-forward network.
[0098] Output: Normalized feature representation, which reduces the difference between layers and helps stabilize model training.
[0099] Processing process: Normalize the output of the feedforward network so that the mean of the output is 0 and the variance is 1. The formula is: Where xx is the output of the feedforward network, sub is the output of the residual connection, σ is the variance calculation, and ∈ is a small constant to prevent division by 0.
[0100] Residual Connection:
[0101] Input: The output of the layer normalization layer and the feature representation before the feed-forward network (i.e. the output of the multi-head self-attention layer).
[0102] Output: Add the output of the layer normalization layer and the original feature representation to form a residual connection to solve the gradient disappearance problem in deep networks.
[0103] Processing process: The residual connection is realized by simply adding the original input and the processed features, that is: out = LN(x) + x, where x is the output of the multi-head self-attention layer and LN(x) is the result after layer normalization and feedforward network processing.
[0104] Output Layer:
[0105] Input: Final output after repeated stacking.
[0106] Output: Perform final output conversion according to task requirements (such as classification, segmentation, etc.).
[0107] Processing: Use the fully connected layer and softmax layer to generate the final prediction results.
[0108] Finally, the output results of the Bin-Parser model are organized and combined with the training corpus to output semantic segmentation and annotated text.
[0109] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.
Claims
1. A cross-CPU architecture binary file semantic segmentation and entity recognition method for trust innovation adaptation, characterized by: The following steps are involved: Step 1: Data collection and annotation; Step 2: Load and preprocess the data; Step 3: construct a binary file segment classification model, and classify the loaded data using the binary file segment classification model; Step 4: Input the classified data into the segment loading module for processing; Step 5: Input the data processed by the segment loading module into the Bin-Parser model and GloVe vector embedding respectively. The output of the Bin-Parser model is combined with the GloVe model training corpus generated by GloVe vector embedding to output semantic segmentation and annotated text.
2. According to claim 1, the cross-CPU architecture binary file semantic segmentation and entity recognition method for information innovation adaptation is characterized by: The specific implementation method of step 1 is: collect a variety of ARM and x86, RISC-V binary files, including application, library and operating system binary files; these files are used as input files, and the pyelftools tool is used to automatically annotate the key semantic units and entities in the ELF file, including functions, variables, segments, opcodes, operands, symbol tables and character tables.
3. According to claim 1, the cross-CPU architecture binary file semantic segmentation and entity recognition method for information innovation adaptation is characterized by: The specific implementation method of step 2 is: use the pyelftools tool to parse the collected binary files, and extract the required content according to the structure of the binary files: Step 2.1, parse ELF file: use pyelftools to parse ELF file and extract detailed information of each part; Step 2.2, Generate annotation data: Generate detailed annotation data for each part, including the label of each byte; Step 2.3, Save the annotation data: Save the annotation data in JSON format for subsequent use.
4. According to claim 1, the cross-CPU architecture binary file semantic segmentation and entity recognition method for information innovation adaptation is characterized by: The binary file segment classification model constructed in the step 3 includes: a 1-dimensional convolution layer, a pooling layer, an LSTM layer, a fully connected layer and a final fully connected layer, wherein the input passes through the convolution layer and the ReLU activation function, and then through the pooling layer, the input passes through the LSTM layer, the output of the last time step is taken, and the output of the LSTM passes through the fully connected layer and the ReLU activation function; the global information is combined with the local features, and the classification result is output through the final fully connected layer, and the classification result ELF file constitutes the classification code.
5. According to claim 1, the cross-CPU architecture binary file semantic segmentation and entity recognition method for information innovation adaptation is characterized by: The step 4 comprises the following steps: Step 4.1, binary file segment classification model result conversion: get the processing result from the binary file segment classification model, and calculate the starting position and offset size of each segment according to the processing result; Step 4.2, binary data reading: read the binary data from the starting position and offset obtained above according to the binary format; Step 4.3, data distribution: Distribute the read data to the GloVe vector embedding and Bin-Parser models respectively.
6. The cross-CPU architecture binary file semantic segmentation and entity recognition method for information innovation adaptation according to claim 1 is characterized in that: The processing method of GloVe vector embedding in step 5 is: read and parse the ELF file using the libelf library, perform actions similar to word segmentation, and organize them into separate function calls and code segments for training to build a corpus; use the instruction set specification and ABI specification to establish a mapping between binary and instruction set, and build a co-occurrence matrix of Glove; use the collected expected information to train the Glove model, and the Glove model saves the word embedding vector that reflects the semantics of the binary file; save the Glove model as a file.