Efficient linker design method based on multi-thread collaborative optimization

Abstract

The invention relates to an efficient linker design method based on multi-thread collaborative optimization. Firstly, dynamic task division and priority queue scheduling are adopted, the link process is decomposed into a plurality of task units capable of being executed in parallel, and the computing power of a multi-core processor is fully utilized. Secondly, a hierarchical symbol table is introduced, hot symbols are stored in a lockless hash table, cold data are stored in a read-write lock protected red-black tree, thread competition is effectively reduced, and efficient concurrent access is achieved; and meanwhile, a sectional relocation table and a progress counter supported by atomic operation are adopted, so that the concurrency performance is further improved. And finally, generating an intermediate file with a dependency label through a compiler, pre-obtaining a symbol dependency relationship and a dependency graph between modules, avoiding repeated analysis, and designing a dynamic feedback mechanism, so that the linker can feed back optimization data to the compiler to form a closed-loop optimization system. The method is suitable for large-scale projects and high-performance computing scenes, and the resource utilization rate is optimized while the link speed is increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer software optimization, and particularly relates to a collaborative optimization method for a compiler and a linker, and more particularly to an efficient linker design method based on multi-threading and concurrent data structures. Background Art

[0002] In the process of software development, the linker is a key tool for merging the object files generated by the compiler into an executable file. With the continuous expansion of the scale of software projects and the increasing demand for high-performance computing, traditional linkers face the problem of low efficiency when dealing with large-scale projects. Especially in the context of the popularization of multi-core processors, traditional linkers fail to make full use of the multi-core computing power, resulting in a long link process time and becoming a performance bottleneck in the software development process.

[0003] Traditional linkers usually adopt a single-threaded design and process tasks such as symbol resolution, relocation calculation, and target file generation in sequence. When dealing with large-scale projects, due to the dependency relationship between tasks and the serial execution mode, this design cannot make full use of the parallel computing power of multi-core processors. In addition, the design of data structures such as symbol tables and relocation tables is not optimized for concurrent access, resulting in thread competition and performance degradation.

[0004] In the prior art, the optimization of linkers mainly focuses on single-thread performance improvement and memory management optimization. For example, reducing the link time by improving the data structure of the symbol table or optimizing the relocation algorithm. However, these methods do not fundamentally solve the parallel computing problem in a multi-core environment. In addition, the collaborative optimization between the compiler and the linker has not been fully emphasized, resulting in repeated parsing and resource waste.

[0005] In terms of concurrent data structures, the prior art usually uses global locks or read-write locks to protect shared data. Although data consistency can be guaranteed, significant thread competition overhead will be introduced. Especially in data structures such as symbol tables with high-frequency access, thread competition seriously affects the concurrent performance of the linker.

[0006] In the traditional linker design, the link process usually adopts single-threaded execution and processes tasks such as symbol resolution, relocation calculation, and target file generation in sequence. When dealing with large-scale projects, due to the dependency relationship between tasks and the serial execution mode, this design cannot make full use of the parallel computing power of multi-core processors. In addition, the design of data structures such as symbol tables and relocation tables is not optimized for concurrent access, resulting in thread competition and performance degradation. Glossary of Terms:

[0007] Linker: A tool that combines the object files generated by the compiler into an executable file, responsible for symbol resolution, relocation calculation, and generating the final executable file.

[0008] Multithreading: A concurrent execution technique that creates multiple threads to execute tasks simultaneously, making full use of the computing power of multi-core processors.

[0009] Dynamic task partitioning: According to the computational complexity and dependencies of tasks, tasks are decomposed into multiple sub-tasks that can be executed in parallel to improve execution efficiency.

[0010] Priority queue scheduling: A task scheduling strategy that determines the execution order according to the priority of tasks, ensuring that high-priority tasks are executed first.

[0011] Concurrent data structure: A data structure that supports concurrent access by multiple threads, protected by lock-free design or read-write locks, reducing thread contention and improving concurrency performance.

[0012] Lock-free hash table: A concurrent data structure that achieves efficient concurrent access through lock-free algorithms and is suitable for scenarios with high-frequency access.

[0013] Read-write lock: A concurrency control mechanism that allows multiple threads to read data simultaneously, but requires exclusive access for writing, and is suitable for scenarios with more reads than writes.

[0014] Segmented relocation table: Divide the relocation table into multiple segments, and each segment is processed by an independent thread to improve concurrency performance.

[0015] Atomic operation: An indivisible operation that ensures the atomicity of operations in a multi-threaded environment and avoids data contention.

[0016] Compiler-linker collaborative optimization: By generating intermediate files with dependency annotations through the compiler, obtaining symbol dependencies in advance, avoiding repeated parsing by the linker, and designing a dynamic feedback mechanism to form a closed-loop optimization system.

[0017] Dynamic feedback mechanism: Monitor performance data in real time during the linking process and feedback optimization suggestions to the compiler for optimization adjustment in subsequent compilation phases. Summary of the Invention

[0018] The purpose of the present invention is to propose an efficient linker design method based on multi-threaded collaborative optimization to improve the linking efficiency and reduce memory occupancy, which is applicable to large-scale projects and high-performance computing scenarios.

[0019] An efficient linker design method based on multi-threaded collaborative optimization proposed by the present invention includes dynamic task partitioning, designing a hierarchical symbol table, designing a segmented relocation table, collaborative optimization between the compiler and the linker, link execution, performance monitoring and feedback. The specific steps are as follows: (1) Dynamic task partitioning According to the computational complexity and dependency relationship of tasks, the link process of the linker is decomposed into multiple sub-task units that can be executed in parallel, supporting multi-threaded concurrent execution; the sub-task units can be symbol resolution, relocation calculation, and object file generation; and through the priority queue scheduling method, the sub-task units are dynamically allocated to idle threads to ensure that high-priority tasks are executed first; make full use of the computing power of multi-core processors, create a thread pool, and initialize multiple worker threads. Each worker thread obtains sub-tasks from the task queues of multiple sub-task units that can be executed in parallel and executes them to improve the link efficiency; (2) Designing a hierarchical symbol table Create a hierarchical symbol table, store hot symbols in a lock-free hash table, and store cold data in a red-black tree protected by a read-write lock to reduce thread contention and achieve efficient concurrent access; (3) Designing a segmented relocation table Adopt a segmented relocation table and a progress counter supported by atomic operations, divide the relocation table into multiple segments, each segment is processed by an independent thread, and synchronize the execution progress of each thread through the progress counter supported by atomic operations to further improve the concurrent performance; (4) Collaborative optimization between the compiler and the linker Generate an intermediate file with dependency annotations through the compiler, pre-obtain symbol dependencies and the dependency graph between modules to avoid repeated parsing; design a dynamic feedback mechanism to enable the linker to feedback optimization data to the compiler during operation to form a closed-loop optimization system; the linker records performance data such as symbol access frequency and thread load during operation and feedbacks optimization suggestions to the compiler for adjusting the symbol generation and dependency annotation strategies in subsequent compilation stages; (5) Link execution Based on the collaborative optimization in step (4), start a multi-threaded link process, dynamically schedule the sub-task units in step (1), and concurrently access the hierarchical symbol table in step (2) and the segmented relocation table in step (3) to complete the generation of the object file; (6) Performance monitoring and feedback Monitor performance data in real time during the link process and feedback optimization suggestions to the compiler for optimization adjustment in subsequent compilation stages.

[0020] In the present invention, the method is applicable to large-scale projects and high-performance computing scenarios, while improving the link speed, reducing memory occupancy, and optimizing resource utilization.

[0021] In the present invention, the compiler and the linker can be GCC, LLVM, etc. When dealing with large-scale projects, it can significantly shorten the linking time and reduce the memory occupancy, and is applicable to the fields of high-performance computing and software development. Through the separate management of control information and data information, the present invention realizes efficient linking and resource optimization, providing strong technical support for software development.

[0022] In the present invention, during the operation of the linker in step (6), it monitors performance data in real time, and the performance data includes symbol access frequency, thread load, etc.

[0023] The beneficial effects of the present invention are as follows: The present invention realizes efficient linking through the following innovative designs: 1. Multi-threaded task partitioning and scheduling: The linking process is decomposed into multiple task units that can be executed in parallel, and a dynamic task partitioning and priority queue scheduling strategy is adopted to make full use of the computing power of multi-core processors to improve the linking efficiency. 2. Concurrent data structure design: A hierarchical symbol table is introduced, with hot symbols stored in a lock-free hash table and cold data stored in a red-black tree protected by a read-write lock, reducing thread contention and achieving efficient concurrent access; at the same time, a segmented relocation table and a progress counter supported by atomic operations are adopted to further improve the concurrent performance. 3. Compiler and linker collaborative optimization: The compiler generates intermediate files with dependency annotations to pre-obtain symbol dependency relationships and inter-module dependency graphs, avoiding repeated parsing; a dynamic feedback mechanism is designed to enable the linker to feedback optimization data to the compiler during operation, forming a closed-loop optimization system.

[0024] The present invention realizes efficient linking and resource optimization, providing strong technical support for software development. Starting from the parallel computing ability and data structure design of the linker, the present invention solves the performance bottleneck problem of traditional linkers in multi-core environments, is applicable to large-scale projects and high-performance computing scenarios, significantly improves the linking efficiency and reduces the memory occupancy. Therefore, the present invention is different from the prior art and has significant technical advantages. In view of the above problems, the present invention proposes an efficient linker design method based on multi-threaded collaborative optimization. Through dynamic task partitioning and priority queue scheduling, the linking process is decomposed into multiple task units that can be executed in parallel, making full use of the computing power of multi-core processors. At the same time, concurrent data structures such as hierarchical symbol tables and segmented relocation tables are introduced to reduce thread contention and improve the concurrent access efficiency. In addition, through the collaborative optimization of the compiler and the linker, symbol dependency relationships are pre-obtained to avoid repeated parsing, and a dynamic feedback mechanism is designed to form a closed-loop optimization system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the index structure of the present invention.

[0026] Figure 2 Schematic diagram of collaborative optimization of the present invention.

[0027] Figure 3 Schematic diagram of multi-threaded optimization of the present invention.

[0028] Figure 4 Flowchart of the present invention. Detailed implementation manners

[0029] The present invention will be further described below with reference to the accompanying drawings through embodiments.

[0030] Embodiment 1: The present invention is an efficient linker design method based on multi-threaded collaborative optimization, applicable to large-scale software projects and high-performance computing scenarios. Taking the GCC compiler on the Linux platform as an example, the specific implementation manners of the present invention will be described in detail below.

[0031] As Figures 1-4 shown, the specific steps are as follows: (1) Optimize the memory layout and data structure of the linker. First, decompose the linking process into subtasks such as symbol resolution, relocation calculation, and object file generation, and support multi-threaded concurrent execution. Adopt dynamic task partitioning and priority queue scheduling strategies to dynamically allocate tasks to idle threads to ensure that high-priority tasks are executed first. To make full use of the computing power of multi-core processors, create a thread pool and initialize multiple worker threads, and each thread obtains tasks from the task queue and executes them. (2) In terms of concurrent data structure design, introduce a hierarchical symbol table, store hot symbols in a lock-free hash table to support efficient concurrent access; store cold data in a red-black tree protected by a read-write lock to reduce thread contention.

[0032] (3) Adopt a segmented relocation table, divide the relocation table into multiple segments, and each segment is processed by an independent thread, and synchronize the execution progress of each thread through a progress counter supported by atomic operations. These optimized designs significantly reduce thread contention and improve concurrent performance.

[0033] (4) In terms of collaborative optimization between the compiler and the linker, generate intermediate files with dependency annotations through the compiler, pre-obtain symbol dependency relationships and module dependency graph information, and avoid repeated parsing by the linker. In addition, design a dynamic feedback mechanism to enable the linker to feedback performance optimization data to the compiler during operation for adjusting the symbol generation and dependency relationship annotation strategies in subsequent compilation stages, forming a closed-loop optimization system.

[0034] (5) In the link execution stage, the symbol table is parsed concurrently by multiple threads, and the parsing results are stored in the hierarchical symbol table; the segmented relocation table is processed concurrently by multiple threads to calculate and update the address information of the target file; the target file is generated concurrently by multiple threads and the result is written to disk. The entire link process is executed concurrently by multiple threads, significantly shortening the link time.

[0035] (6) In terms of performance monitoring and feedback, the linker monitors performance data such as symbol access frequency and thread load in real time during operation, and feeds the monitoring data back to the compiler for optimization and adjustment in subsequent compilation stages. This dynamic feedback mechanism ensures continuous improvement of the optimization strategy and further improvement of performance.

[0036] Through the implementation of the present invention, the execution efficiency and resource utilization rate of the linker have been significantly improved. The concurrent execution of multiple threads makes full use of the computing power of multi-core processors, the optimization of concurrent data structures reduces thread competition, and the collaborative optimization of the compiler and the linker avoids repeated parsing and reduces memory occupancy. The present invention is applicable to large-scale software projects and high-performance computing scenarios. Especially in a multi-core processor environment, it can significantly improve the link efficiency and reduce memory occupancy, providing strong technical support for software development.

Claims

1. An efficient linker design method based on multi-threaded collaborative optimization, characterized in that It includes dynamic task partitioning, designing a hierarchical symbol table, designing a segmented relocation table, collaborative optimization of the compiler and linker, link execution, performance monitoring and feedback. The specific steps are as follows: (1) Dynamic task partitioning According to the computational complexity and dependency relationships of tasks, the linker's linking process is decomposed into multiple sub-task units that can be executed in parallel, supporting multi-threaded concurrent execution; The sub-task units can be symbol resolution, relocation calculation, and object file generation; and through the priority queue scheduling method, the sub-task units are dynamically allocated to idle threads to ensure that high-priority tasks are executed first; making full use of the computing power of multi-core processors, a thread pool is created and multiple worker threads are initialized. Each worker thread fetches and executes sub-tasks from the task queues of multiple sub-task units that can be executed in parallel to improve the linking efficiency; (2) Designing a hierarchical symbol table Create a hierarchical symbol table, store hot symbols in a lock-free hash table, and store cold data in a red-black tree protected by a read-write lock to reduce thread contention and achieve efficient concurrent access; (3) Designing a segmented relocation table Adopt a segmented relocation table and a progress counter supported by atomic operations, divide the relocation table into multiple segments, each segment is processed by an independent thread, and the execution progress of each thread is synchronized through a progress counter supported by atomic operations to further improve the concurrent performance; (4) Collaborative optimization of the compiler and linker Generate an intermediate file with dependency annotations through the compiler, pre-obtain symbol dependency relationships and the dependency graph between modules to avoid repeated parsing; design a dynamic feedback mechanism so that the linker feeds optimization data back to the compiler during operation to form a closed-loop optimization system; During operation, the linker records performance data such as symbol access frequencies and thread loads, and feeds optimization suggestions back to the compiler for adjusting symbol generation and dependency relationship annotation strategies in subsequent compilation stages; (5) Link execution Based on the collaborative optimization in step (4), start a multi-threaded linking process, dynamically schedule the sub-task units in step (1), and concurrently access the hierarchical symbol table in step (2) and the segmented relocation table in step (3) to complete the generation of object files; (6) Performance monitoring and feedback During the linking process, monitor performance data in real time and feed optimization suggestions back to the compiler for optimization and adjustment in subsequent compilation stages.

2. The efficient linker design method based on multi-threaded collaborative optimization according to claim 1, characterized in that The method is applicable to large-scale projects and high-performance computing scenarios, improving the linking speed while reducing memory occupancy and optimizing resource utilization.

3. An efficient linker design method based on multi-threaded collaborative optimization according to claim 1, characterized in that The compiler and linker use GCC and LLVM.

4. An efficient linker design method based on multi-threaded collaborative optimization according to claim 1, characterized in that In step (6), the linker monitors performance data in real time during operation, and the performance data includes symbol access frequencies and thread loads.

Images