Multi-mode-based hardware accelerator design space search method and device and electronic equipment

Abstract

The invention discloses a hardware accelerator design space searching method and device based on multiple modalities and electronic device.The method comprises the steps that an original design code and a design space file are obtained, and a first design scheme set is determined according to the original design code and the design space file; determining a third design scheme set from the first design scheme set based on a hierarchical evaluation model; and determining analogue simulation data corresponding to each third design scheme in the third design scheme set, and determining a target design scheme from the third design scheme set according to the analogue simulation data. Based on the technical scheme, a strategy of combining a lightweight rapid evaluation model and a high-precision complex evaluation model is utilized, modular disassembly is performed on a design space, layer-by-layer exploration optimization is performed, and an optimal design scheme is found in an automatic manner, so that the design efficiency is remarkably improved, the design cost is reduced, and the design efficiency is improved. And a new solution is provided for the optimization problem of the complex FPGA design.

Description

Technical Field

[0001] This invention relates to the field of automated design technology, and in particular to a multimodal hardware accelerator design space search method, apparatus, and electronic device. Background Technology

[0002] With the rapid development of machine learning technology, its influence is gradually spreading to the field of electronic design automation (EDA). More and more cutting-edge technologies are being used in various stages of EDA, covering key aspects such as high-level synthesis, logic synthesis, and layout planning. This field, which previously relied heavily on human experience and traditional methods, is about to undergo a profound transformation driven by machine learning.

[0003] In the high-level synthesis stage, machine learning methods can analyze and transform hardware behavior described by high-level languages ​​to obtain data flow graphs, text sequences, and abstract syntax trees, which better map abstract algorithm logic onto hardware structures. Then, through machine learning representation learning methods, a large number of design instances are learned to predict the performance of different designs, thereby efficiently and automatically identifying the optimal hardware implementation scheme.

[0004] However, the existing HLS optimization process involves engineers writing HLS code that conforms to specifications, carefully designing and inserting compiler directives, and determining a series of design parameters such as loop unrolling factors and pipeline methods. Then, tools such as VivadoHLS / Vitis are used for advanced synthesis to generate register-transfer level code and a relatively accurate preliminary synthesis report. Engineers then modify parameter configurations based on the information in the report and their personal experience, and perform advanced synthesis again to continuously optimize. It is evident that the existing optimization process heavily relies on the designer's accumulated design experience, requires engineers to possess complex hardware knowledge, and necessitates repeated trials and manual adjustments. Furthermore, the development cycle can last for several weeks, making it time-consuming, labor-intensive, and inefficient, failing to meet the demands of a rapidly iterating market. Summary of the Invention

[0005] This invention provides a multimodal hardware accelerator design space search method, device, electronic device, and storage medium. By constructing a multimodal hierarchical representation method, feature extraction technology, and candidate design point evaluation model, it enables in-depth analysis and efficient exploration of HLS design code and its design space parameters.

[0006] According to one aspect of the present invention, a multimodal hardware accelerator design space search method is provided, comprising:

[0007] Obtain the original design code and design space file, and determine the first design scheme set based on the original design code and the design space file;

[0008] A third set of design schemes is determined from the first set of design schemes based on a hierarchical evaluation model, wherein the hierarchical evaluation model includes a lightweight evaluation model and a deep evaluation model;

[0009] The simulation data corresponding to each third design scheme in the set of third design schemes is determined, and the target design scheme is determined from the set of third design schemes based on the simulation data.

[0010] According to another aspect of the present invention, a multimodal hardware accelerator-based space search device is provided, comprising:

[0011] The first design scheme set determination module is used to obtain the original design code and the design space file, and determine the first design scheme set based on the original design code and the design space file;

[0012] The third design scheme set determination module is used to determine the third design scheme set from the first design scheme set based on the hierarchical evaluation model, wherein the hierarchical evaluation model includes a lightweight evaluation model and a deep evaluation model.

[0013] The target design scheme determination module is used to determine the simulation data corresponding to each third design scheme in the third design scheme set, and to determine the target design scheme from the third design scheme set based on the simulation data.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multimodal hardware accelerator design space search method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the multimodal hardware accelerator design space search method according to any embodiment of the present invention.

[0019] The technical solution of this invention involves acquiring original design code and design space files, determining a first design scheme set based on the original design code and the design space files, determining a third design scheme set from the first design scheme set based on a hierarchical evaluation model, further determining simulation data corresponding to each third design scheme in the third design scheme set, and determining a target design scheme from the third design scheme set based on the simulation data. Based on the above technical solution, a strategy combining a lightweight, fast evaluation model and a high-precision, complex evaluation model is used to modularly decompose the design space and explore optimization layer by layer, finding the optimal design scheme in an automated manner. This significantly improves design efficiency, reduces design costs, and provides a new solution to the optimization challenges of complex FPGA designs.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the accompanying drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a multimodal hardware accelerator design space search method provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the design space search strategy provided in an embodiment of the present invention;

[0024] Figure 3 This is a flowchart of a hardware accelerator design space search method based on multimodality provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the MRMH model framework provided in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the MRMH-CA model framework provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the MRMH-MoE model framework provided in an embodiment of the present invention;

[0028] Figure 7This is a structural block diagram of a space search device based on a multimodal hardware accelerator design, provided in an embodiment of the present invention.

[0029] Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to the process, method, product, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart illustrating a multimodal hardware accelerator design space search method provided in an embodiment of the present invention. This embodiment is applicable to situations where design schemes are optimized during the design process of a hardware accelerator. This method can be executed by a multimodal hardware accelerator design space search device, which can be implemented in hardware and / or software. This multimodal hardware accelerator design space search device can be configured in an electronic device, such as a server or terminal device. Figure 1 As shown, the method includes:

[0034] S110. Obtain the original design code and design space file, and determine the first design scheme set based on the original design code and the design space file.

[0035] The original design code can be a pre-coded code file corresponding to the hardware accelerator. The design space file can describe all valid combinations of adjustable parameters corresponding to the code file, as well as subspace types corresponding to the design code. Subspace types can include loop subspaces, storage subspaces, and interface subspaces, etc. The first design scheme set can be a collection of first design schemes obtained by combining the design code and adjustable parameters.

[0036] Specifically, the process involves acquiring the original design code and design space file, and determining the first design scheme set based on these. It's important to note that the design space encompasses all effective combinations of adjustable parameters during the design process. For example, in an HLS design scenario, different pipeline stages, loop unrolling factors, and data storage methods combine to form the design space. Each HLS design instance can be considered a sampling point within this vast design space. After acquiring the HLS code and design space definition file, in-depth analysis of the HLS code and design space parameters is conducted. The code can be partitioned according to pre-set subspace partitioning rules. This partitioning process fully considers the structural characteristics of the code, such as function modules, loop structures, and data dependencies, decomposing the design space into multiple relatively independent and easily explorable subspaces. This modular decomposition effectively reduces the complexity of design space exploration. For instance, it can be based on long-accumulated prior knowledge, automatically partitioning the code and its corresponding design space, or it can be done manually.

[0037] S120. Determine the third design scheme set from the first design scheme set based on the hierarchical evaluation model.

[0038] The hierarchical evaluation model includes a lightweight evaluation model and a deep evaluation model. The hierarchical evaluation model is used to evaluate the first design scheme. The third design scheme set can be a collection of design schemes that need to be simulated and verified, obtained by screening from the first design scheme set.

[0039] Specifically, determining the third design scheme set from the first design scheme set based on the hierarchical evaluation model can be achieved by determining the evaluation score corresponding to each first design scheme in the first design scheme set according to the hierarchical evaluation model, and then determining the third design scheme set from the first design scheme set based on the evaluation score.

[0040] Based on the above technical solution, the step of determining the third design scheme set from the first design scheme set based on the hierarchical evaluation model includes: determining the lightweight evaluation result corresponding to each first design scheme in the first design scheme set according to the lightweight evaluation model; determining the second design scheme set from the first design scheme set based on the lightweight evaluation result; and determining the third design scheme set from the second design scheme set according to the deep evaluation model.

[0041] The lightweight evaluation model can be used to quickly evaluate the first design scheme. The lightweight evaluation results can be understood as evaluation parameters used to quantify the performance of each design scheme. The second design scheme set can be a collection of second design schemes obtained after preliminary screening of the first design scheme.

[0042] Specifically, a depth-first search algorithm is used to traverse each sub-design space. Lightweight models within the hierarchical evaluation model, such as the unimodal HARP model or the lightweight MRMH model, are employed to rapidly evaluate a massive number of design points. Lightweight models offer advantages such as low computational complexity and fast inference speed, enabling quantitative analysis of design points in a short time. Based on the evaluation results, and according to preset performance indicators such as processing latency and resource consumption, the design points are sorted in descending or ascending order to select the top M candidate design points with high potential, significantly narrowing the scope of subsequent evaluations. In the high-precision deep evaluation stage, the Top M second-stage design schemes selected in the previous stage are input into the high-precision model for secondary evaluation. A comprehensive and in-depth analysis of the design points is performed, resulting in more accurate performance prediction values. Based on the prediction values, the design points are sorted again to obtain the Top N third-stage design schemes.

[0043] Based on the above technical solution, the step of determining the third design scheme set from the second design scheme set according to the deep evaluation model includes: for each second design scheme in the second design scheme set, determining the design code corresponding to the second design scheme; determining the multimodal feature set corresponding to the design code; determining the deep evaluation result corresponding to each second design scheme according to the multimodal feature set; and determining the third design scheme set from the second design scheme set based on the deep evaluation result.

[0044] The multimodal feature set can be a collection of code features corresponding to the design code. The multimodal feature set includes graph representation features, code representation features, and structural representation features. The deep evaluation result can be an evaluation score used to assess the performance of the design scheme. The design code can be code obtained by combining the original code with the parameters of the current scheme.

[0045] Specifically, for each second design scheme in the second design scheme set, the design code corresponding to the second design scheme is determined, and the multimodal feature set corresponding to the design code is determined. Based on the multimodal feature set, a deep evaluation result corresponding to each second design scheme is determined, and the third design scheme set is determined from the second design scheme set based on the deep evaluation result. For example, for each second design scheme in the second design scheme set, it needs to be converted into design code. The design code can cover the key information of the design scheme in a computer-processable form, such as the description of the structural layout, component parameters, and functional logic of the design scheme. A multimodal feature set is extracted from the design code. Multimodal features can describe the characteristics of the design code from different angles and of different types. For example, they may include text features, structural features, performance features, etc. Features from different modalities are combined to form a feature set that comprehensively reflects the characteristics of the design code. The deep evaluation model is used to process the multimodal feature set to obtain the deep evaluation result corresponding to each second design scheme. It should be noted that the deep evaluation model is pre-trained and can comprehensively and deeply evaluate the design scheme based on the input multimodal feature set and by considering various feature factors. The evaluation results can be quantitative scores or qualitative evaluation levels, used to measure the quality of design schemes in multiple dimensions. Based on the in-depth evaluation results, design schemes that meet the conditions are selected from the second set of design schemes to form the third set of design schemes. This can be done by retaining design schemes with evaluation results higher than a pre-set threshold, or by sorting the evaluation results and selecting the top-ranked design schemes.

[0046] Based on the above technical solution, determining the multimodal feature set corresponding to the design code includes: compiling the design code using a code compilation tool to obtain intermediate code, and converting the intermediate code into a graph data structure to be processed; encoding the source language information of the design code into the graph data structure to be processed, and constructing high-level pseudo-nodes to obtain graph representation features corresponding to the design code.

[0047] The high-level pseudo-nodes are used to aggregate local information from the intermediate code. The multimodal feature set is determined based on the graph representation features and the design code. The code compilation tool can be a tool used to compile HLS design code. The intermediate code can be code obtained by compiling the design code.

[0048] Specifically, the LLVM intermediate code compiled from the native HLS design can be processed using the open-source ProGraML tool and transformed into a corresponding ProgramGraph object. This object describes the nodes and connections of the graph structure data corresponding to the LLVM intermediate code. Further encoding the basic information from the LLVM intermediate code and the optimization primitives from the native HLS design into the graph yields graph structure data containing complete HLS design information, which can then be used for training and inference of the graph neural network predictor. To address the long-range dependency problem, high-level pseudo-node abstractions must be constructed, with each pseudo-node representing a block of the intermediate representation. This method compresses detailed information from local subgraphs into representative abstract features. Secondly, the message passing mechanism of the graph neural network allows for long-distance information interaction between pseudo-nodes, alleviating the oversmoothing limitations of traditional GNNs. Moreover, the information from lower-level nodes is aggregated to higher-level nodes through message propagation, significantly reducing the computational complexity of subsequent cross-attention modal interactions. Furthermore, pragma nodes and their parameters can be directly applied to pseudo-nodes to transform the feature vectors of the entire intermediate representation block, decoupling functional design and optimization parameters to obtain a detailed feature representation of the HLS design.

[0049] Based on the above technical solution, determining the multimodal feature set based on the graph representation features and the design code includes: dividing the design code into blocks based on preset stop words and the number of segmentation lemmas to determine code representation features corresponding to the design code; processing the design code according to preset prompt statements and a code understanding big model to obtain structural information in a preset format, and determining structural representation features based on the structural information; and constructing the multimodal feature set based on the graph representation features, the code representation features, and the structural representation features.

[0050] The preset stop words can be a pre-set stop word list, and the number of segmentation units can be a pre-set number of word segments. Pre-set prompts can be understood as prompts used to instruct the code understanding model to parse the code; corresponding prompts can be set according to requirements. The code understanding model can be a large language model used to parse the code; the required large model can be selected according to needs.

[0051] Specifically, for frequently occurring specific terms in code sequences, such as variable names (e.g., result_buf, index), function names (e.g., compute_kernel()), and compiler directives (e.g., #pragmaHLS), a domain-specific stop word list can be constructed, which can effectively improve the accuracy of subsequent semantic analysis. Furthermore, in code sequence processing, since code sequences generally contain long texts, with an average length exceeding 500 tokens, a fixed-length chunking strategy can be adopted to divide the code sequence into chunks of 64 tokens each. By introducing a bidirectional sliding window technique, each code chunk retains 12 tokens of contextual information from its preceding and following chunks. Through this fixed-window contextual information, the model can accurately locate the position of each code chunk within the entire code sequence, deeply understand its role in the overall logic, and thus form code representation features corresponding to the designed code. Specifically, the bidirectional sliding window mechanism allows the model to not only focus on the content of a code chunk itself when analyzing it, but also better combine the information from its preceding and following chunks, thereby grasping the overall semantics of the code more comprehensively and accurately. This full utilization of contextual information avoids the semantic fragmentation problems that may result from simple block processing, ensuring that the model can understand the logical structure and functional intent of the code as a whole. Furthermore, by using prompts to drive code understanding, the large model extracts structured information from the code sequence in HTML format, including function names, function parameters, loop structures, and pragma statements. Based on this, a Python script is designed to parse the obtained HTML file and extract important structural information. Subsequently, the extracted code structure sequence is block-based according to its order, nesting relationships, and pragma statements, thus forming a high-level representation of the code sequence modality. When this is aggregated with the source code sequence, it can simultaneously bridge remote dependencies and decouple compilation guidance information from the original code.

[0052] S130. Determine the simulation data corresponding to each third design scheme in the third design scheme set, and determine the target design scheme from the third design scheme set based on the simulation data.

[0053] Specifically, such as Figure 2As shown, a hierarchical evaluation model stack is used for in-depth exploration. A lightweight, rapid evaluation phase employs a depth-first search (DFS) algorithm to systematically traverse each sub-design space. Lightweight models within the hierarchical evaluation model stack, such as the unimodal HARP model or the lightweight MRMH model, are used to rapidly evaluate a massive number of design points. Lightweight models offer advantages such as low computational complexity and fast inference speed, enabling quantitative analysis of design points in a short time. Based on the evaluation results, and according to preset performance indicators such as processing latency and resource consumption, the design points are sorted in descending or ascending order, selecting the top M candidate design points with high potential, significantly narrowing the scope of subsequent evaluations.

[0054] The high-precision in-depth evaluation stage inputs the TopM design points selected in the previous stage into the high-precision model for secondary evaluation. A comprehensive and in-depth analysis of the design points is performed, resulting in more accurate performance predictions. Based on these predictions, the design points are ranked again to obtain the TopN high-quality design points, providing high-value samples for subsequent hardware verification.

[0055] Based on the above technical solution, the step of determining the simulation data corresponding to each of the third design schemes in the third design scheme set includes: determining the design subspace corresponding to the design space file; and determining the simulation data corresponding to each design subspace according to the simulation software and the third design scheme.

[0056] In this context, the design subspace can be understood as the parameter space corresponding to the functional units that correspond to the hardware mapping semantics. The design subspace includes at least one of the following types: loop subspace, storage subspace, and interface subspace.

[0057] Specifically, during the hardware synthesis and verification phase, for the N final selected design points, a professional HLS toolchain, such as VivadoHLS, is invoked to conduct hardware synthesis and simulation verification. Performance data of the design in a real hardware environment, including clock frequency and resource usage, is obtained. Based on the measured data, design schemes are compared and analyzed, and the optimal performance design is selected. The corresponding subspace parameters, such as loop unrolling factor and pipeline stages, are then fixed as the optimal configuration for the current exploration layer, providing a reference for subsequent exploration. When the subspaces generated by the current partitioning strategy have been explored, but the entire design space has not yet been covered (i.e., there are still untouched outer spaces), a dynamic expansion strategy is activated. The core of this strategy is to first fix the parameters of the explored subspaces; stable parameter configurations provide a solid foundation for subsequent exploration. Subsequently, expansion proceeds towards the outer structure of the design. For example, at the code structure level, it delves into higher-level modules, which often undertake more complex system functions; or it moves towards higher-level loop structures, where loops typically control a wider range of data processing flows. In this way, new unexplored subspaces are identified and partitioned. During implementation, the above steps are repeated continuously: exploring newly defined subspaces. When subspace exploration is complete but full space coverage is not yet achieved, dynamic expansion is initiated again. This iterative process, like ripples, gradually expands the exploration scope of the design space, spreading from the initial local area to the entire design space. Once all design space parameters are determined, the parameters of all subspaces are systematically merged to obtain the optimal design scheme that meets the design objectives.

[0058] The technical solution of this invention involves acquiring original design code and design space files, determining a first design scheme set based on the original design code and the design space files, determining a third design scheme set from the first design scheme set based on a hierarchical evaluation model, further determining simulation data corresponding to each third design scheme in the third design scheme set, and determining a target design scheme from the third design scheme set based on the simulation data. Based on the above technical solution, a strategy combining a lightweight, fast evaluation model and a high-precision, complex evaluation model is used to modularly decompose the design space and explore optimization layer by layer, finding the optimal design scheme in an automated manner. This significantly improves design efficiency, reduces design costs, and provides a new solution to the optimization challenges of complex FPGA designs.

[0059] Example 2

[0060] Figure 3This is a flowchart illustrating a multimodal hardware accelerator design space search method provided in an embodiment of the present invention. This embodiment further optimizes the technical solution described above by determining the depth evaluation result corresponding to each second design scheme based on the multimodal feature set. For example... Figure 3 As shown, it includes:

[0061] S310. For each second design scheme in the second design scheme set, determine the design code corresponding to the second design scheme;

[0062] S320. Determine the multimodal feature set corresponding to the design code, input the multimodal feature set into the multimodal design evaluation model, and obtain the depth evaluation result output by the multimodal design evaluation model.

[0063] S330. Based on the depth evaluation results, determine the third design scheme set from the second design scheme set and determine the third design scheme set from the second design scheme set according to the depth evaluation model.

[0064] The multimodal design evaluation model is a pre-trained neural network model. The multimodal design evaluation model includes a low-level hybrid expert model and a high-level expert model; the high-level expert model is used to determine the weight coefficients corresponding to the low-level hybrid expert model; the low-level hybrid expert model includes at least a node-level hybrid expert model, a pseudo-node-level hybrid expert model, and a code block-level hybrid expert model; the high-level expert model determines the deep evaluation result based on the weight coefficients and the output of the low-level hybrid expert model.

[0065] It should be noted that, in order to construct the multimodal candidate design point evaluation module, in-depth research was conducted on cutting-edge multimodal technologies. During this research, it was discovered that BLIP2 cleverly utilizes contrastive learning and matching learning for cross-modal model alignment and ingeniously designs a Q-Former structure. This architecture designs mini-batch learnable vectors as a medium between image and text modalities to achieve multimodal interaction and obtain the final representation vector. Although this application scenario does not involve images but rather controls the data flow graph, and the text modality is the code sequence of HLS design, it still shares similarities with the BLIP2 method. Based on this, an initial multimodal candidate design point evaluation model was constructed and named MRMH (Multimodal Representation Learning Model for HLS), as follows... Figure 4 As shown.

[0066] The multimodal design evaluation model of this invention comprises two sub-modules sharing a self-attention layer, referred to as the GraphTransformer and the CodeTransformer, respectively, which share the self-attention layer. During the operation of the MRMH model, learnable vectors play a crucial role as a key medium for information interaction. On one hand, learnable vectors interact with the text modality through the shared self-attention module, capturing the rich semantics inherent in the text modality.

[0067] On the other hand, learnable vectors, through the cross-attention module of the graph transformer, efficiently interact with high-level graph nodes (pseudo-nodes) in the hierarchical graph representation, mining topological and attribute information from the graph structure data. Through this dual-path information interaction, the model can comprehensively integrate information from different modalities. To achieve interaction between different modalities while reducing computational cost, a batch of learnable vectors is randomly initialized using Gaussian at the model input, serving as a medium for interaction between the graph modality and the code / text modality. Gaussian random initialization assigns reasonable and dispersed initial values ​​to the vectors, avoiding slow model convergence or getting trapped in local optima due to improper initial value settings. Graph modalities and code modalities carry different types of key information.

[0068] The graph modality visually reflects the control data flow in HLS design through the topological structure of nodes and edges, demonstrating the direction of data flow and control logic. The code modality, on the other hand, uses textual form to contain rich semantic information, detailing the functionality and logical structure of the code.

[0069] During propagation, small batches of random vectors extract key information from the code modality through a self-attention mechanism, integrating and absorbing information from the code model. Then, through a cross-attention mechanism, they interact with nodes in the graph neural network, breaking down information barriers between modalities. This allows the model to comprehensively understand HLS from various perspectives, improving model performance.

[0070] The task of Graph-Code Matching (GCM) is to determine whether the current control data flow graph and the code semantically match; essentially, it is a binary classification problem. The control data flow graph is used to obtain a feature map G∈R. B×D Code modalities (including code sequences and captions) generate contextual embeddings C∈R through a frozen language model. L×D It should be noted that D is the dimension, K is the query dimension, B is the number of nodes, and L is the length. D and K are manually set hyperparameters. In graph code matching learning, the learnable query and the code modality can pay attention to each other, employing bimodal attention. Finally, based on the learnable query, Q′∈R is output. K×DThe matching degree is obtained through a fully connected layer, and the loss function is as follows:

[0071]

[0072] Where y i ∈{0,1} represents the matching label, while p i This represents the predicted probability output by the model, where N is the batch size.

[0073] The goal of Graph-Code Contrastive Learning (GCC) is to map images and text to the same embedding space, making the representations of matching pairs closer and those of non-matching pairs further apart. It learns the encodings of queries and codes within each pair, preventing mutual attention and thus avoiding information leakage. For each query q′∈Q′ in each cross-modal output, the cosine similarity with the code modal output C is calculated. The query with the highest similarity is selected as the final similarity s′. Furthermore, bidirectional comparisons are performed, calculating the similarity from graph to code and the similarity from code to graph to enhance symmetry. The loss function is as follows:

[0074]

[0075] Where s i The figure code represents the cosine similarity, and τ = 0.07 is the temperature parameter.

[0076] After pre-training, a candidate design point evaluation model is trained on a labeled dataset using mean squared error as the loss function. The formula is as follows:

[0077] The hierarchical graph modality uses pseudo-nodes as a high-level graph representation, while the code modality uses the code structure sequence (Caption) proposed in this invention as a high-level representation. Therefore, the block features of the code structure sequence and the pseudo-node modality of the graph modality can interact directly through a cross-attention layer, thus deepening the interaction between the graph and code modalities without excessively increasing computational cost. Furthermore, the feature extraction module still uses the CodeT5 and graph neural network architecture from MRMH.

[0078] Based on the above analysis, a new model architecture was designed, such as... Figure 5 As shown, because it uses a cross-attention mechanism to directly perform intermodal interactions, it is called MRMH-CA.

[0079] Both code sequences and code structure sequences were divided into blocks. For single-modality graph modalities, information propagation was performed using a graph neural network. For code modalities, whether code or caption, the first [CLS] of each block was selected as the block's feature. Information exchange between code blocks and caption blocks was achieved through a bidirectional self-attention mechanism in the Transformer layer. The formula is as follows:

[0080]

[0081] For cross-modal information interaction, the Caption block features of the code modality semantic information are integrated as a high-level representation of the code modality. Information is extracted through interaction with pseudo-nodes of the graph modality high-level representation via Cross-Att (cross-attention mechanism), alleviating the problem of high computational cost of the cross-attention mechanism and enhancing the model's adaptability and flexibility in handling complex tasks. The calculation formula for the cross-modal cross-attention mechanism is as follows.

[0082]

[0083] After the Caption block extracts the graph modal information, it is processed by the feedforward layer and passed to the next block. Then, information is exchanged through Bi-Self-Attention, thereby passing the graph information captured by the CaptionToken to other code sequence blocks.

[0084] A deep analysis of the MRMH-CA framework reveals that, although divided into three parts, its core architecture is highly similar to MRMH, primarily consisting of two sub-modules: graph neural networks and Transformers. In the hardware design space exploration process, accurately predicting the performance and quality of candidate design points is crucial for improving design efficiency and optimizing hardware performance. Therefore, integrating beneficial features from multiple modalities becomes a key step.

[0085] MRMH-CA performs pooling on the output of the graph neural network to obtain graph modality feature vectors after passing through a PragmaasMLP layer and an additional GNN layer. The PragmaasMLP layer, as a key component of the graph encoder, effectively encodes the pragma nodes connected to the pseudo-nodes and their parameters, achieving decoupling and interaction between code and instruction statements, greatly enriching the graph modality features. After this layer, a further message passing layer in the graph neural network results in an even richer feature representation for the overall nodes.

[0086] The Transformer substructure leverages the powerful capabilities of its cross-attention layer to achieve deep fusion of graph and code modal information. Specifically, the cross-attention mechanism dynamically allocates the intensity of information interaction by calculating attention weights between features from different modalities, thereby effectively capturing fine-grained correlations between modalities. Pooling is also applied to the output of the Transformer substructure to extract the fused cross-modal features.

[0087] The graph modal feature vector is concatenated with cross-modal features to form a comprehensive feature vector containing rich information. This vector is then fed into a multi-head multilayer perceptron. The multi-head multilayer perceptron performs nonlinear transformations on the comprehensive features through multiple linear transformation layers and activation functions, uncovering complex relationships between features and ultimately achieving accurate prediction of the performance and quality of candidate design points.

[0088] Because MRMH uses learnable vectors for cross-modal interaction, it's necessary to consider whether the parameters of the previously obtained Transformer structure can be directly used in MRMH-CA. Since MRMH-CA also involves cross-modal interaction, theoretically, the parameters accumulated by the Transformer structure in previous learning contain an understanding of inter-modal relationships and could be directly reused. However, subtle differences in model structure may lead to poor parameter reuse performance; therefore, a comprehensive evaluation of parameter applicability is essential.

[0089] In the previous MRMH model construction, GCM (Graph-Code Matching) and GCC (Graph-Code Contrastive Learning) strategies were used to effectively bridge the gap between different modalities. Among them, GCM has a high similarity to the current model architecture. Since it does not require a single-peak attention mechanism, only minor adjustments are needed when adapting it to the current model.

[0090] Now let's analyze Graph-Code Contrastive Learning (GCC). In GCC, learnable vectors and code tokens cannot follow each other to avoid information leakage. For code encoding, as long as it doesn't pass through a cross-attention layer during feedforward, the learnable vectors serve to extract graph modal information. This means that similarity calculations can be performed directly using the features of the learnable vectors and code features to perform contrastive learning, which is essentially an update of the cross-attention layer, aligning the latent feature spaces of the graph and the code.

[0091] Based on the above considerations, implementing cross-attention (GCC) inevitably leads to information leakage; therefore, GCC is performed between the graph module and the pure code module. By implementing GCC between these two modules, not only can the parameters of subsequent layers of HARP be effectively updated, but it also helps to align the semantic spaces of the graph and code modalities, improving the model's ability to understand cross-modal information. The Transformer part continues to use the existing parameters of the MRMH model. This approach ensures the model's stability when handling complex data while avoiding the high computational cost of retraining.

[0092] After the pre-training phase is completed, the model enters a new training phase, namely, targeted training of the multi-head perceptron. This aims to further explore the complex relationships between features, optimize the model's ability to represent various types of information, and obtain a candidate design point evaluation model.

[0093] Low-level expert models include node-level hybrid expert models, which are built at the level of ordinary nodes in graph modalities. This model can dynamically allocate computing resources for different types of ordinary nodes and learn node-specific detailed features; pseudo-node-level hybrid expert models, which utilize the abstract characteristics of pseudo-nodes to enable the model to learn the local features and relationships of the graph structure more efficiently; and code block-level hybrid expert models, which, by leveraging the semantic integrity of code blocks, can better capture the functionality and logic of code fragments. The following sections will detail each low-level expert model and its working principles.

[0094] After encoding with a 6-layer graph neural network (GNN), the node-level hybrid expert model (NPT) encodes the compiler instruction nodes connected to the pseudo-nodes and their parameters into vector form. These vectors are then integrated into the vector representation of the pseudo-nodes through a multilayer perceptron layer. Next, a single-layer graph neural network is used to propagate the compiler instruction information captured by the pseudo-nodes. This layer plays a crucial role in determining the degree to which each node is affected by the current compiler instruction configuration. Therefore, n graph neural network layers are constructed at this layer, each acting as an expert. The computation process is as follows:

[0095]

[0096] Among them, h i W represents the embedding vector of the node before the update. G The parameters are those of a gating network with dimension n×d. The gating network determines the weights of each expert based on the embedding vector of the current node, where n is the number of experts. After each graph neural network expert completes message passing, the updated node embedding vector h′ is obtained by calculating a weighted sum. iIn the hierarchical representation of a graph, pseudo-nodes can aggregate local information and facilitate long-distance information interaction. Therefore, a hybrid expert model is constructed for pseudo-nodes to enhance their expressive power. n linear layers are designed as the expert model to transform the pseudo-node embeddings. The specific calculation process is as follows:

[0097]

[0098] Similar to Node Hybrid Experts (NodeMoE), where expert weights are determined based on node embeddings, the update operation here is performed only on pseudo-nodes, and the expert model is replaced by linear layers instead of graph neural network layers.

[0099] A code block-level hybrid expert model. Similar to pseudo-nodes, block-level embeddings also possess the ability to aggregate information and facilitate long-distance information interaction. The Caption block can interact with pseudo-nodes and capture rich code information through self-attention layers. n different self-attention layers were constructed as the expert model to enhance its expressive power. The computation process is as follows:

[0100]

[0101] Among them, h i W represents the block-level embedding vector before the update. G The dimension n×d is the parameter of the gating network. The gating network determines the weight of each expert based on the embedding vector of the current word, where n is the number of experts. After each attention expert performs a bidirectional attention operation, a new block embedding vector h′ is obtained by calculating a weighted sum. i .

[0102] In real-world scenarios, different kernels exhibit vastly different characteristics, leading to significant performance differences among the three proposed expert models when processing different kernels. To effectively integrate the strengths of each expert model across different kernels and optimize overall prediction performance, a hierarchical hybrid expert (MoE) architecture was designed. For example... Figure 6 As shown, a high-level MoE model is built to summarize and integrate the results generated by the low-level MoE model. The specific calculation process is as follows:

[0103] g = Concate(g1, g2, g3)

[0104]

[0105] Where g1, g2, and g3 represent graph embedding vectors obtained from three expert models, and W G It is a high-level gated network.

[0106] The three graph embedding vectors are concatenated, and a gating network is used to determine the weight of each low-level expert. Then, the weighted sum of their prediction results is calculated to obtain the final prediction result.

[0107] It's important to note that because a two-layer hybrid expert (MoE) model is constructed, the granularity of the lower-layer hybrid expert models differs, leading to differences in computational cost and convergence speed. For example, the computational cost of a node-level hybrid expert operating on all nodes is significantly higher than that applied to a hybrid expert operating on a few selected pseudo-nodes. This difference in convergence causes the higher-level expert models to tend to assign more weights to models with faster convergence, resulting in other lower-level expert models losing their intended role. To address this issue, the predictor training process on the dataset is divided into two stages. In the first stage, several rounds of warm-up training are performed, during which the three lower-layer hybrid expert (MoE) models are trained independently to narrow the performance gap between them. In the second stage, the entire model is trained end-to-end. The loss function is as follows:

[0108]

[0109] in, It is the MSE loss function. L Rh This is the regularization term for the high-level expert model. L R These are the regularization terms for the three low-level MoE models, and their expressions are as follows:

[0110] The technical solution of this invention integrates graph modalities, code sequence modalities, and code structure sequence (Caption) modalities to form a hierarchical multimodal representation system. This method can comprehensively capture key information in HLS design, improving the accuracy and effectiveness of evaluation. Data from different modalities provides different perspectives and information; by fusing this information, the essence of the design can be understood more comprehensively. For text-formatted data, a Transformer model is used to capture long sequence dependencies. This helps improve the efficiency and accuracy of feature extraction. Graph neural networks excel at handling complex graph structures, while Transformer models perform exceptionally well when handling sequence data. Combining the two can effectively improve adaptability to different data types. Finally, an automatic design space exploration method based on prior knowledge is proposed. This method, based on a divide-and-conquer strategy and a progressive model collaboration mechanism, can significantly improve the efficiency of design space exploration without sacrificing design quality. This method uses a refined design space partitioning strategy and a depth-first search algorithm to perform hierarchical design space exploration of each subspace block, which can quickly filter low-quality design points and further evaluate high-quality options. In summary, this invention addresses the problems of insufficient evaluation accuracy and low exploration efficiency in existing technologies by introducing multimodal data representation, efficient feature extraction models, and an automatic design space exploration method based on prior knowledge, thus providing a new and efficient solution for automatic design space exploration in hardware accelerators.

[0111] Example 3

[0112] Figure 7 This is a structural block diagram of a space search device based on a multimodal hardware accelerator design, provided by an embodiment of the present invention. Figure 7 As shown, the device includes: a first design scheme set determination module 710, a third design scheme set determination module 720, and a target design scheme determination module 730; wherein,

[0113] The first design scheme set determination module 710 is used to obtain the original design code and the design space file, and determine the first design scheme set based on the original design code and the design space file;

[0114] The third design scheme set determination module 720 is used to determine the third design scheme set from the first design scheme set based on the hierarchical evaluation model, wherein the hierarchical evaluation model of the design schemes includes a lightweight evaluation model and a deep evaluation model.

[0115] The target design scheme determination module 730 is used to determine the simulation data corresponding to each third design scheme in the third design scheme set, and to determine the target design scheme from the third design scheme set based on the simulation data.

[0116] Based on the above technical solution, the third design scheme set determination module is used to determine the lightweight evaluation result corresponding to each first design scheme in the first design scheme set according to the lightweight evaluation model; determine the second design scheme set from the first design scheme set based on the lightweight evaluation result; and determine the third design scheme set from the second design scheme set according to the deep evaluation model.

[0117] Based on the above technical solution, the third design scheme set determination module is used to determine the design code corresponding to each second design scheme in the second design scheme set; determine the multimodal feature set corresponding to the design code; determine the depth evaluation result corresponding to each second design scheme based on the multimodal feature set, wherein the multimodal feature set includes graph representation features, code representation features and structural representation features; and determine the third design scheme set from the second design scheme set based on the depth evaluation result.

[0118] Based on the above technical solution, the third design scheme set determination module is used to input the multimodal feature set into the multimodal design evaluation model to obtain the deep evaluation result output by the multimodal design evaluation model, wherein the multimodal design evaluation model is a pre-trained neural network model.

[0119] Based on the above technical solution, the multimodal design evaluation model includes a low-level hybrid expert model and a high-level expert model; the high-level expert model is used to determine the weight coefficients corresponding to the low-level hybrid expert model; the low-level hybrid expert model includes at least a node-level hybrid expert model, a pseudo-node-level hybrid expert model, and a code block-level hybrid expert model; the high-level expert model determines the deep evaluation result based on the weight coefficients and the output of the low-level hybrid expert model.

[0120] Based on the above technical solution, the third design scheme set determination module is used to compile the design code using a code compilation tool to obtain intermediate code, and convert the intermediate code into a graph data structure to be processed; encode the source language information of the design code into the graph data structure to be processed, and construct high-level pseudo-nodes to obtain graph representation features corresponding to the design code, wherein the high-level pseudo-nodes are used to aggregate local information of the intermediate code; and determine the multimodal feature set based on the graph representation features and the design code.

[0121] Based on the above technical solution, the third design scheme set determination module is used to divide the design code into blocks based on the number of preset stop words and segmentation units, and determine the code representation features corresponding to the design code; process the design code according to preset prompt statements and code understanding big model to obtain structural information in a preset format, and determine structural representation features based on the structural information; and construct the multimodal feature set based on the graph representation features, the code representation features and the structural representation features.

[0122] The technical solution of this invention involves acquiring original design code and design space files, determining a first design scheme set based on the original design code and the design space files, determining a third design scheme set from the first design scheme set based on a hierarchical evaluation model, further determining simulation data corresponding to each third design scheme in the third design scheme set, and determining a target design scheme from the third design scheme set based on the simulation data. Based on the above technical solution, a strategy combining a lightweight, fast evaluation model and a high-precision, complex evaluation model is used to modularly decompose the design space and explore optimization layer by layer, finding the optimal design scheme in an automated manner. This significantly improves design efficiency, reduces design costs, and provides a new solution to the optimization challenges of complex FPGA designs.

[0123] The hardware accelerator design space search device based on multimodality provided in this embodiment of the invention can execute the hardware accelerator design space search method based on multimodality provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0124] Example 4

[0125] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0126] like Figure 8As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0127] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0128] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as multimodal hardware accelerator design space search methods.

[0129] In some embodiments, the multimodal hardware accelerator design space search method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal hardware accelerator design space search method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multimodal hardware accelerator design space search method by any other suitable means (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. The computer program may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer program may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0135] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0136] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multimodal hardware accelerator design space search method, characterized in that, include: Obtain the original design code and design space file, and determine the first design scheme set based on the original design code and the design space file; A third set of design schemes is determined from the first set of design schemes based on a hierarchical evaluation model, wherein the hierarchical evaluation model includes a lightweight evaluation model and a deep evaluation model; The simulation data corresponding to each third design scheme in the set of third design schemes is determined, and the target design scheme is determined from the set of third design schemes based on the simulation data.

2. The method according to claim 1, characterized in that, The process of determining the third design scheme set from the first design scheme set based on the hierarchical evaluation model includes: Based on the lightweight evaluation model, determine the lightweight evaluation result corresponding to each first design scheme in the first design scheme set; Based on the lightweight assessment results, a second design scheme set is determined from the first design scheme set, and a third design scheme set is determined from the second design scheme set according to the deep assessment model.

3. The method according to claim 2, characterized in that, The step of determining the third design scheme set from the second design scheme set according to the deep evaluation model includes: For each second design scheme in the second design scheme set, determine the design code corresponding to the second design scheme; A multimodal feature set corresponding to the design code is determined, and a depth evaluation result corresponding to each second design scheme is determined based on the multimodal feature set, wherein the multimodal feature set includes graph representation features, code representation features, and structural representation features; The third design scheme set is determined from the second design scheme set based on the in-depth evaluation results.

4. The method according to claim 3, characterized in that, The step of determining the depth evaluation result corresponding to each second design scheme based on the multimodal feature set includes: The multimodal feature set is input into the multimodal design evaluation model to obtain the deep evaluation result output by the multimodal design evaluation model, wherein the multimodal design evaluation model is a pre-trained neural network model.

5. The method according to claim 4, characterized in that, The multimodal design evaluation model includes a low-level hybrid expert model and a high-level expert model; the high-level expert model is used to determine the weight coefficients corresponding to the low-level hybrid expert model. The low-level hybrid expert model includes at least a node-level hybrid expert model, a pseudo-node-level hybrid expert model, and a code block-level hybrid expert model; the high-level expert model determines the depth evaluation result based on the weight coefficients and the output of the low-level hybrid expert model.

6. The method according to claim 3, characterized in that, The determination of the multimodal feature set corresponding to the design code includes: The design code is compiled using a code compilation tool to obtain intermediate code, which is then converted into a graph data structure to be processed. The source language information of the design code is encoded into the graph data structure to be processed, and high-level pseudo-nodes are constructed to obtain graph representation features corresponding to the design code. The high-level pseudo-nodes are used to aggregate local information of the intermediate code. The multimodal feature set is determined based on the graph representation features and the design code.

7. The method according to claim 5, characterized in that, Determining the multimodal feature set based on the graph representation features and the design code includes: The design code is divided into blocks based on the preset stop words and the number of segmentation units to determine the code representation features corresponding to the design code; Based on preset prompts and code, the large model is understood to process the design code, obtain structural information in a preset format, and determine structural representation features based on the structural information; The multimodal feature set is constructed based on the graph representation features, the code representation features, and the structural representation features.

8. The method according to claim 1, characterized in that, The determination of the simulation data corresponding to each third design scheme in the set of third design schemes includes: Determine the design subspace corresponding to the design space file, wherein the design subspace includes at least one of the following: loop subspace, storage subspace, and interface subspace; The simulation data corresponding to each design subspace is determined based on the simulation software and the third design scheme.

9. A space search device based on a multimodal hardware accelerator design, characterized in that, include: The first design scheme set determination module is used to obtain the original design code and the design space file, and determine the first design scheme set based on the original design code and the design space file; The third design scheme set determination module is used to determine the third design scheme set from the first design scheme set based on the hierarchical evaluation model, wherein the hierarchical evaluation model includes a lightweight evaluation model and a deep evaluation model. The target design scheme determination module is used to determine the simulation data corresponding to each third design scheme in the third design scheme set, and to determine the target design scheme from the third design scheme set based on the simulation data.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multimodal hardware accelerator design space search method according to any one of claims 1-8.

Images