Joint search method and device of neural network and hardware

Abstract

The present invention provides a method and apparatus for joint search of neural networks and hardware, which are applied to the field of computer technology and are used to address defects in accuracy assessment and hardware performance search in related technologies. The method comprises: sampling the neural network search space corresponding to the current iteration to obtain G neural networks to be evaluated corresponding to the current iteration, and determining the accuracy score value of the gth neural network to be evaluated; performing hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine the hardware performance value of the gth neural network to be evaluated; determining the adaptability value of the gth neural network to be evaluated based on the ratio between the accuracy score value and the hardware performance value of the gth neural network to be evaluated, and determining a candidate neural network corresponding to the current iteration from the G neural networks to be evaluated; and determining a target neural network suitable for performing a target processing task from the candidate neural networks corresponding to each of the multiple iterations.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method and device for joint search of a neural network and hardware. Background Art

[0002] While neural networks excel in various fields, they also increase computational complexity and design difficulty. Therefore, combining neural network architecture search with hardware accelerator optimization has become a research hotspot.

[0003] Neural network architecture search techniques in related technologies have shortcomings in accuracy assessment and hardware performance search. Specifically, existing methods rely on over-parameterized supernet training, resulting in significant computational resource and time overhead for computers and other service devices. Regarding hardware performance search, related methods rely on single-core architectures for hardware performance search, failing to fully consider the differences and computational preferences of network layers. This results in insufficient hardware performance accuracy and slow computational speeds for computers and other service devices. Summary of the Invention

[0004] In view of the above problems, the present invention provides a joint search method and device of a neural network and hardware.

[0005] According to a first aspect of the present invention, a joint search method for a neural network and hardware is provided, which is applied to a computing device, comprising: sampling a neural network search space corresponding to a current iteration in a storage unit based on preset neural network search conditions to obtain G neural networks to be evaluated corresponding to the current iteration; performing a gradient calculation task using a computing unit based on a g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration to obtain a target gradient value set for the g-th neural network to be evaluated, where G ≥ 1 and 1 ≤ g ≤ G; determining a precision score value for the g-th neural network to be evaluated based on the target gradient value set for the g-th neural network to be evaluated; performing hardware performance simulation on hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine a hardware performance value for the g-th neural network to be evaluated; determining an adaptability value for the g-th neural network to be evaluated based on a ratio between the precision score value and the hardware performance value of the g-th neural network to be evaluated to obtain adaptability values ​​for the G neural networks to be evaluated; determining the neural network to be evaluated corresponding to the highest adaptability value among the G neural networks to be evaluated as a candidate neural network corresponding to the current iteration; and determining a target neural network suitable for executing the target processing task from the candidate neural networks corresponding to each of the multiple rounds.

[0006] According to an embodiment of the present invention, the object processing task includes at least one of the following: an object detection task, a text processing task, and an image classification task.

[0007] According to an embodiment of the present invention, for a g-th neural network to be evaluated among G neural networks to be evaluated corresponding to a current round of iteration, determining a target gradient value set for the g-th neural network to be evaluated includes: determining M batches of training label data input to the g-th neural network to be evaluated, where M≥1; for an m-th batch of training label data among the M batches, determining a loss value for the m-th batch of the g-th neural network to be evaluated based on the m-th batch of training label data of the g-th neural network to be evaluated, where 1≤m≤M; respectively calculating the loss value of the m-th batch and derivatives of each of N weight parameters of the g-th neural network to be evaluated to obtain N gradient values ​​for the m-th batch, and obtaining M×N gradient values ​​for the M batches, where N≥1; and screening the M×N gradient values ​​to obtain a target gradient value set for the g-th neural network to be evaluated.

[0008] According to an embodiment of the present invention, M×N gradient values ​​are screened to obtain a target gradient value set for a g-th neural network to be evaluated, including: for an n-th weight parameter among N weight parameters of the g-th neural network to be evaluated, the M gradient values ​​corresponding to the n-th weight parameter are divided into intervals to obtain multiple gradient intervals, 1≤n≤N; based on the density of each gradient interval, the gradient interval with the largest density among the multiple gradient intervals is selected as the n-th target gradient interval corresponding to the n-th weight parameter to obtain N target gradient intervals; based on the n-th target gradient interval, the n-th target gradient value set corresponding to the n-th weight parameter is determined to obtain N target gradient value sets; and the N target gradient value sets are used as the target gradient value set for the g-th neural network to be evaluated.

[0009] According to an embodiment of the present invention, determining the accuracy score value of the g-th neural network to be evaluated based on the target gradient value set of the g-th neural network to be evaluated includes: determining the gradient average of the n-th target gradient value set for the g-th neural network to be evaluated; determining the gradient variance value of the n-th target gradient value set; determining the n-th initial accuracy score value based on the ratio between the gradient average and the gradient variance value, to obtain N initial accuracy score values; and determining the accuracy score value of the g-th neural network to be evaluated based on the N initial accuracy score values.

[0010] According to an embodiment of the present invention, a heterogeneous multi-core architecture includes multiple hardware cores; hardware performance simulation is performed on the hardware configuration parameters of the neural network to be evaluated using the heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated, including: mapping the g-th neural network to be evaluated to multiple hardware cores to obtain multiple target hardware cores; determining the target number of computing units and target bandwidth of each target hardware core based on the ratio of the computing demand and the bandwidth demand of each target hardware core; optimizing the data flow parameters of each target hardware core to obtain the target data flow parameters of each target hardware core; using a performance simulation framework, hardware performance simulation is performed on the target number of computing units, target bandwidth and target data flow parameters of each target hardware core to determine the hardware performance value of the g-th neural network to be evaluated.

[0011] According to an embodiment of the present invention, the g-th neural network to be evaluated includes multiple operators; mapping the g-th neural network to be evaluated to multiple hardware cores to obtain multiple target hardware cores, including: clustering all operators according to the dimension parameters of each operator to obtain the category of each operator; mapping operators with the same category to the same hardware core to obtain multiple target hardware cores.

[0012] According to an embodiment of the present invention, the data flow parameters of each target hardware core are optimized to obtain the target data flow parameters of each target hardware core, including: for the data flow parameters of any target hardware core, the data flow parameters are iteratively optimized to obtain candidate data flow parameters; and the candidate data flow parameters that meet the preset iteration conditions of the data flow parameters are determined as the data flow parameters.

[0013] According to an embodiment of the present invention, before optimizing the data flow parameters of each target hardware core, the method further includes: pruning the data flow parameters of any target hardware core to obtain pruned data flow parameters.

[0014] A second aspect of the present invention provides a joint search device for a neural network and hardware, comprising: a storage unit for storing a neural network search space; a computing unit configured to: sample the neural network search space corresponding to the current iteration in the storage unit based on a preset neural network search condition to obtain G neural networks to be evaluated corresponding to the current iteration; perform a gradient calculation task based on the g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration to obtain a target gradient value set for the g-th neural network to be evaluated, G≥1, 1≤g≤G; determine the accuracy score value of the g-th neural network to be evaluated based on the target gradient value set of the g-th neural network to be evaluated; a heterogeneous multi-core architecture configured to: perform hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated to determine the hardware performance value of the g-th neural network to be evaluated; wherein, The computing unit is further configured to: determine the adaptability value of the g-th neural network to be evaluated based on the ratio between the accuracy score value and the hardware performance value of the g-th neural network to be evaluated, and obtain the adaptability values ​​of G neural networks to be evaluated; determine the neural network to be evaluated corresponding to the highest adaptability value among the adaptability values ​​of the G neural networks to be evaluated as the candidate neural network corresponding to the current round of iteration; and determine a target neural network suitable for executing the target processing task from the candidate neural networks corresponding to each of the multiple rounds.

[0015] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0017] The fifth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0018] According to an embodiment of the present invention, based on preset neural network search conditions, the neural network search space corresponding to the current round of iteration in the storage unit is sampled to obtain G neural networks to be evaluated corresponding to the current round of iteration; based on the g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current round of iteration, the computing unit is used to perform a gradient calculation task to obtain a target gradient value set of the g-th neural network to be evaluated; based on the target gradient value set of the g-th neural network to be evaluated, the accuracy score value of the g-th neural network to be evaluated is determined; hardware performance simulation is performed on the hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated; based on the ratio between the accuracy score value and the hardware performance value of the g-th neural network to be evaluated, the adaptability value of the g-th neural network to be evaluated is determined to obtain the adaptability values ​​of the G neural networks to be evaluated; the neural network to be evaluated corresponding to the highest adaptability value among the G neural networks to be evaluated is determined as the candidate neural network corresponding to the current round of iteration; and a target neural network suitable for executing the target processing task is determined from the candidate neural networks corresponding to multiple rounds. Since the computing unit is used to perform the gradient calculation task, the relationship between the gradient information and the model convergence of the neural network to be evaluated is fully utilized when determining the accuracy score value corresponding to the neural network to be evaluated, thereby avoiding the design of a heavy and complex supernet, reducing the computing overhead of service equipment such as computers, and improving the search accuracy of the target neural network while ensuring the improvement of the computing speed of service equipment such as computers; in addition, since the hardware performance of the neural network to be evaluated is simulated by relying on a heterogeneous multi-core architecture, the problem of a single-core architecture failing to fully consider the differences and computing preferences of the network layers is avoided, so that service equipment such as computers does not need to reconfigure the hardware performance evaluation architecture for different neural networks to be evaluated, thereby ensuring the stable operation of service equipment such as computers and reducing the storage space occupied by service equipment such as computers. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 A diagram schematically illustrates an application scenario of a method and device for joint search of a neural network and hardware according to an embodiment of the present invention.

[0021] Figure 2 The flowchart of the joint search method of the neural network and hardware according to an embodiment of the present invention is schematically shown.

[0022] Figure 3 A schematic diagram of a heterogeneous multi-core architecture according to an embodiment of the present invention is schematically shown.

[0023] Figure 4The figure schematically shows data flow parameters according to an embodiment of the present invention.

[0024] Figure 5 A schematic diagram of a joint search architecture of a neural network and hardware according to an embodiment of the present invention is schematically shown.

[0025] Figure 6 The figure schematically shows the test results of the hardware searcher according to an embodiment of the present invention.

[0026] Figure 7 The structure block diagram of the joint search device of the neural network and hardware according to an embodiment of the present invention is schematically shown.

[0027] Figure 8 The figure schematically shows a block diagram of an electronic device suitable for implementing a joint search method of a neural network and hardware according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] In related joint search techniques, accuracy information is mostly obtained through training-based or predictor-based methods. After generating different neural networks, each needs to be fully trained to evaluate its accuracy. By integrating all subnetworks in the design space into an over-parameterized supernet, the supernet only needs to be trained once, avoiding the need to repeatedly train each subnetwork. Specifically, the subnetworks can inherit weights from the pre-trained supernet, allowing for rapid accuracy information acquisition. Predictor-based methods, on the other hand, predict model accuracy by learning the accuracy performance of the network structure. Therefore, these methods typically require collecting a batch of actual training accuracy data for neural networks, forming a dataset containing network structures and their corresponding accuracy. Based on this dataset, a deep learning model is trained to predict the potential accuracy of different network structures. However, these training-based neural network ranking methods face significant computational resource and time overhead. Even a one-time training method based on a supernet can still require over 100 GPU hours. Furthermore, when the search space changes, the supernet design or predictor training preparation must be redesigned, resulting in significant reconstruction costs and limited versatility.

[0033] To obtain hardware performance, hardware search-based work requires searching for the most suitable hardware configuration parameters for a certain designed network structure. The hardware information of the neural network performing inference under these parameters serves as hardware feedback. Since the hardware design is not determined during the joint search process, the determination of hardware design parameters can be regarded as a sub-search problem. For the hardware search space of existing joint search work, the parallel size of the hardware computing array, data reuse size, and classic data flow selection are explored based on the single-core accelerator architecture. However, the above method searches for hardware structures based on a single-core architecture and classic data flow, and fails to fully consider the differences and computing preferences of the network layers, resulting in insufficient hardware adaptability obtained by the search.

[0034] In view of this, an embodiment of the present invention provides a joint search method for neural networks and hardware, which is applied to a computing device, comprising: sampling a neural network search space corresponding to a current round of iteration in a storage unit based on preset neural network search conditions to obtain G neural networks to be evaluated corresponding to the current round of iteration; performing a gradient calculation task using a computing unit based on a g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current round of iteration to obtain a target gradient value set for the g-th neural network to be evaluated, where G≥1 and 1≤g≤G; determining a precision score value for the g-th neural network to be evaluated based on the target gradient value set for the g-th neural network to be evaluated; performing hardware performance simulation on hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine a hardware performance value for the g-th neural network to be evaluated; determining an adaptability value of the g-th neural network to be evaluated based on a ratio between the precision score value and the hardware performance value of the g-th neural network to be evaluated to obtain adaptability values ​​for the G neural networks to be evaluated; determining the neural network to be evaluated corresponding to the highest adaptability value among the G neural networks to be evaluated as a candidate neural network corresponding to the current round of iteration; and determining a target neural network suitable for performing the target processing task from the candidate neural networks corresponding to each of the multiple rounds.

[0035] Figure 1 A diagram schematically illustrates an application scenario of a method and device for joint search of a neural network and hardware according to an embodiment of the present invention.

[0036] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc.

[0037] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0038] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0040] It should be noted that the neural network and hardware joint search method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the neural network and hardware joint search device provided in the embodiment of the present invention can generally be set in the server 105. The neural network and hardware joint search method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the neural network and hardware joint search device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0042] The following will be based on Figure 1 The scene described by Figures 2 to 6 The joint search method of the neural network and hardware in the embodiment of the invention is described in detail.

[0043] Figure 2 The flowchart of the joint search method of the neural network and hardware according to an embodiment of the present invention is schematically shown.

[0044] like Figure 2 As shown, the joint search method of the neural network and hardware in this embodiment includes operations S210 to S270.

[0045] In operation S210, based on a preset neural network search condition, a neural network search space corresponding to the current iteration in a storage unit is sampled to obtain G neural networks to be evaluated corresponding to the current iteration.

[0046] According to an embodiment of the present invention, the preset neural network search conditions may be the user's preference for the neural network, for example, the accuracy requirements that the obtained neural network needs to meet or the performance adaptation preference with the accelerator hardware.

[0047] According to an embodiment of the present invention, a storage unit may be storage hardware such as a solid-state drive or a mechanical hard drive, and is used to store a neural network search space, wherein the neural network search space may be a neural network template and a series of operators of variable types and sizes. The neural network template has a variable network layer and a fixed network layer. The positions of the variable network layers preset in the template can be filled with operators to form a variety of possible neural network architectures. The neural network search space may be a convolutional neural network search space dominated by convolution operators, or a Transformer search space dominated by attention calculations. Based on the preset neural network search conditions, the operators in the neural network search space corresponding to the current iteration are sampled, and operators that are suitable for hardware accelerators are screened. The sampled operators are then filled into the neural network template to obtain G neural networks to be evaluated corresponding to the current iteration.

[0048] According to an embodiment of the present invention, multiple rounds of iterative optimization are performed in a joint search method of a neural network and hardware, wherein the operations of each round are consistent. For any round, it is necessary to sample the neural network search space corresponding to the current round of iteration based on preset neural network search conditions to obtain G neural networks to be evaluated corresponding to the current round of iteration, wherein the preset neural network search conditions do not need to be repeatedly input.

[0049] In operation S220, based on the g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current round iteration, a gradient calculation task is performed using a computing unit to obtain a target gradient value set for the g-th neural network to be evaluated, where G≥1 and 1≤g≤G.

[0050] According to an embodiment of the present invention, the computing unit may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field-programmable gate array (FPGA), etc., depending on the complexity, performance requirements and application scenarios of the relevant tasks.

[0051] According to an embodiment of the present invention, the gradient calculation task may be to calculate the gradient of the neural network to be evaluated based on training data related to the neural network to be evaluated.

[0052] According to an embodiment of the present invention, the gradient of a neural network is calculated via backpropagation and used to update the neural network's weight parameters to minimize the loss function. The higher the consistency of the gradient of the neural network to be evaluated, the lower the probability of oscillation in updating the relevant parameters of the neural network to be evaluated, and the faster the convergence.

[0053] According to an embodiment of the present invention, the present invention uses several small batches of training data to obtain a set of target gradient values ​​of the neural network to be evaluated. Based on the consistency of the target gradient value sets of different batches, the update oscillation probability of the relevant parameters of the neural network to be evaluated can be represented.

[0054] According to an embodiment of the present invention, the number of small batches of training data used by each neural network to be evaluated in each round are consistent.

[0055] In operation S230 , an accuracy score value of the g th neural network to be evaluated is determined based on the target gradient value set of the g th neural network to be evaluated.

[0056] According to an embodiment of the present invention, a gradient consistency evaluation is performed on relevant parameters of the neural network to be evaluated, and the relationship between gradient information and the model convergence of the neural network is fully utilized to obtain the accuracy score value of the neural network to be evaluated. While improving the efficiency of accuracy evaluation, it gets rid of the dependence on the design of a cumbersome and complex supernet, adapts to different evaluation scenarios, reduces the difficulty of accuracy scoring, and thus reduces the system overhead and improves the processing efficiency of the system.

[0057] In operation S240 , a hardware performance simulation is performed on the hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated.

[0058] According to an embodiment of the present invention, the hardware configuration parameters may be data flow parameters, the number of processing elements, and bandwidth limitations.

[0059] According to an embodiment of the present invention, a heterogeneous multi-core architecture can be a processor architecture with multiple hardware cores, each of which has different hardware configuration parameters. Multiple hardware cores can perform hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated in parallel, and can provide a comprehensive hardware performance evaluation for the neural network to be evaluated, so as to select the neural network architecture that is most suitable for the hardware accelerator in the subsequent iterative search process.

[0060] In operation S250 , the adaptability value of the g-th neural network to be evaluated is determined according to the ratio between the accuracy score value and the hardware performance value of the g-th neural network to be evaluated, thereby obtaining the adaptability values ​​of G neural networks to be evaluated.

[0061] According to an embodiment of the present invention, the adaptability value of the neural network to be evaluated is combined with the accuracy score value and hardware performance value of the neural network to be evaluated to reflect the performance differences of different neural networks to be evaluated.

[0062] According to an embodiment of the present invention, the G neural networks to be evaluated in each round perform the above operations respectively to obtain their respective adaptability values.

[0063] In operation S260 , the neural network to be evaluated corresponding to the highest fitness value among the G neural networks to be evaluated is determined as the candidate neural network corresponding to the current iteration.

[0064] According to an embodiment of the present invention, the neural network to be evaluated corresponding to the highest fitness value among the fitness values ​​of the G neural networks to be evaluated is screened to determine the optimal neural network to be evaluated in the current round.

[0065] In operation S270 , a target neural network suitable for performing the target processing task is determined from the candidate neural networks corresponding to the multiple rounds.

[0066] According to an embodiment of the present invention, the target processing task includes at least one of the following: an object detection task, a text processing task, or an image classification task. For example, when the target processing task is an image classification task, the target neural network can be capable of automatically learning key features in an image and have sufficient neural network depth and complexity to enable the target neural network to be suitable for a corresponding computer device (e.g., a computer device such as a hardware accelerator) to perform the image classification task.

[0067] The target neural network is applied to the target processing task by determining the input and output requirements associated with the selected target processing task, selecting a corresponding target neural network based on the input and output requirements, and processing the input data based on the target neural network to complete the target processing task. For example, for an image classification task, the required input image data and the image input and output formats are determined; based on the required input image data and the image input and output formats, the required target neural network is determined; and then, the target neural network is used to process the input image data to complete the image classification task.

[0068] According to an embodiment of the present invention, in a joint search method for neural networks and hardware, the candidate neural networks obtained in the current iteration round are used as optimization criteria to optimize the neural network search space corresponding to the next iteration round. That is, the G neural networks to be evaluated obtained in the next iteration round are different from the G neural networks to be evaluated in the current iteration round. Each iteration screens its own candidate neural network based on its own G neural networks to be evaluated. Among the candidate neural networks in all iteration rounds, the candidate neural network that meets the predetermined iteration criteria is selected and determined as the target neural network.

[0069] According to an embodiment of the present invention, the predetermined iteration condition may be to complete a preset number of iterations, compare the fitness values ​​of the candidate neural networks in each round, and use the candidate neural network corresponding to the highest fitness value as the target neural network. For example, the preset joint search method of the neural network and hardware needs to complete three rounds of iterations. In the first round, candidate neural network A is obtained, and the fitness value of candidate neural network A is 1; in the second round, the neural network search space is optimized based on candidate neural network A to obtain candidate neural network B, and the fitness value of candidate neural network B is 1.5; in the third round, the neural network search space is optimized based on candidate neural network B to obtain candidate neural network C, and the fitness value of candidate neural network C is 2. At this point, three rounds of iterations are completed, among which the fitness value of candidate neural network C obtained in the third round is the highest, and candidate neural network C is determined as the target neural network.

[0070] According to an embodiment of the present invention, the predetermined iteration condition may also be that when the fitness value of a candidate neural network screened from the G neural networks to be evaluated in a certain round reaches a preset threshold, the iteration is stopped and the candidate neural network is determined as the target neural network. For example, the preset threshold is set to 2, and the joint search method of the neural network and hardware is executed. In the first round, candidate neural network A is obtained, and the fitness value of candidate neural network A is 1; in the second round, the neural network search space is optimized based on candidate neural network A to obtain candidate neural network B, and the fitness value of candidate neural network B is 1.5; in the third round, the neural network search space is optimized based on candidate neural network B to obtain candidate neural network C, and the fitness value of candidate neural network C is 2. At this point, the fitness value of candidate neural network C reaches the preset threshold, the iteration is stopped, and candidate neural network C is determined as the target neural network.

[0071] According to an embodiment of the present invention, based on preset neural network search conditions, the neural network search space corresponding to the current round of iteration in the storage unit is sampled to obtain G neural networks to be evaluated corresponding to the current round of iteration; based on the g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current round of iteration, the computing unit is used to perform a gradient calculation task to obtain a target gradient value set of the g-th neural network to be evaluated; based on the target gradient value set of the g-th neural network to be evaluated, the accuracy score value of the g-th neural network to be evaluated is determined; hardware performance simulation is performed on the hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated; based on the ratio between the accuracy score value and the hardware performance value of the g-th neural network to be evaluated, the adaptability value of the g-th neural network to be evaluated is determined to obtain the adaptability values ​​of the G neural networks to be evaluated; the neural network to be evaluated corresponding to the highest adaptability value among the G neural networks to be evaluated is determined as the candidate neural network corresponding to the current round of iteration; and a target neural network suitable for executing the target processing task is determined from the candidate neural networks corresponding to multiple rounds. Since the computing unit is used to perform the gradient calculation task, the relationship between the gradient information and the model convergence of the neural network to be evaluated is fully utilized when determining the accuracy score value corresponding to the neural network to be evaluated, thereby avoiding the design of a heavy and complex supernet, reducing the computing overhead of service equipment such as computers, and improving the search accuracy of the target neural network while ensuring the improvement of the computing speed of service equipment such as computers; in addition, since the hardware performance of the neural network to be evaluated is simulated by relying on a heterogeneous multi-core architecture, the problem of a single-core architecture failing to fully consider the differences and computing preferences of the network layers is avoided, so that service equipment such as computers does not need to reconfigure the hardware performance evaluation architecture for different neural networks to be evaluated, thereby ensuring the stable operation of service equipment such as computers and reducing the storage space occupied by service equipment such as computers.

[0072] According to an embodiment of the present invention, for a g-th neural network to be evaluated among G neural networks to be evaluated corresponding to a current round of iteration, determining a target gradient value set for the g-th neural network to be evaluated includes: determining M batches of training label data input to the g-th neural network to be evaluated, where M≥1; for an m-th batch of training label data among the M batches, determining a loss value for the m-th batch of the g-th neural network to be evaluated based on the m-th batch of training label data of the g-th neural network to be evaluated, where 1≤m≤M; respectively calculating the loss value of the m-th batch and derivatives of each of N weight parameters of the g-th neural network to be evaluated to obtain N gradient values ​​for the m-th batch, and obtaining M×N gradient values ​​for the M batches, where N≥1; and screening the M×N gradient values ​​to obtain a target gradient value set for the g-th neural network to be evaluated.

[0073] According to an embodiment of the present invention, the training label data may include training data and labels corresponding to the training data, wherein the training label data may be M batches of mini-batch data.

[0074] According to an embodiment of the present invention, for the mth batch of training label data among M batches, the training data is input into the neural network to be evaluated, and the loss value of the mth batch of the gth neural network to be evaluated is calculated by comparing the result output by the neural network to be evaluated with the label corresponding to the training data.

[0075] According to an embodiment of the present invention, there are N weight parameters in the neural network to be evaluated. For any batch of training label data, the N weight parameters have corresponding gradient values. Specifically, the gradient values ​​are determined as shown in formula (1). The loss value of the mth batch and the derivatives of the weight parameters of the N gth neural networks to be evaluated are calculated respectively to obtain the N gradient values ​​of the mth batch.

[0076] (1);

[0077] in, Represents the nth weight parameter among N weight parameters, Represents the mth batch of training label data, where , the mth batch of training label data includes b training data x and b labels y, represents the i-th training data among b training data x, represents the i-th label corresponding to the i-th training data in b labels y, Indicates seeking loss, represents the partial derivative, represents the N gradient values ​​of the mth batch, represents the neural network to be evaluated with the input of the i-th training data, Represents the weight parameters of the neural network to be evaluated.

[0078] According to an embodiment of the present invention, based on the N gradient values ​​obtained for the mth batch, M×N gradient values ​​for M batches of the gth neural network to be evaluated can be obtained. Because excessively large or small gradients are not conducive to the convergence of the neural network to be evaluated, blindly using all information may lead to evaluation bias. Therefore, the M×N gradient values ​​are screened, and excessively large or small gradient values ​​are defined as gradient noise. The target gradient value set for the gth neural network to be evaluated is selected.

[0079] According to an embodiment of the present invention, M×N gradient values ​​are screened to obtain a target gradient value set for a g-th neural network to be evaluated, including: for an n-th weight parameter among N weight parameters of the g-th neural network to be evaluated, the M gradient values ​​corresponding to the n-th weight parameter are divided into intervals to obtain multiple gradient intervals, 1≤n≤N; based on the density of each gradient interval, the gradient interval with the largest density among the multiple gradient intervals is selected as the n-th target gradient interval corresponding to the n-th weight parameter to obtain N target gradient intervals; based on the n-th target gradient interval, the n-th target gradient value set corresponding to the n-th weight parameter is determined to obtain N target gradient value sets; and the N target gradient value sets are used as the target gradient value set for the g-th neural network to be evaluated.

[0080] According to an embodiment of the present invention, for the nth weight parameter among the N weight parameters of the gth neural network to be evaluated, the M gradient values ​​corresponding to the nth weight parameter are divided into intervals. For example, 0≤gradient value<1 is the first gradient interval, 1≤gradient value<2 is the second gradient interval, and 2≤gradient value<3 is the third gradient interval.

[0081] According to an embodiment of the present invention, the density of a gradient interval may be the number of gradient values ​​within each interval. The gradient interval with the largest density is selected from multiple gradient intervals as the nth target gradient interval corresponding to the nth weight parameter. For example, if the first gradient interval has 3 gradient values, the second gradient interval has 10 gradient values, and the third gradient interval has 2 gradient values, the second gradient interval is used as the target gradient interval. The 10 gradient values ​​within the target gradient interval are used as the target gradient value set.

[0082] According to an embodiment of the present invention, determining the accuracy score value of the g-th neural network to be evaluated based on the target gradient value set of the g-th neural network to be evaluated includes: determining the gradient average of the n-th target gradient value set for the g-th neural network to be evaluated; determining the gradient variance value of the n-th target gradient value set; determining the n-th initial accuracy score value based on the ratio between the gradient average and the gradient variance value, to obtain N initial accuracy score values; and determining the accuracy score value of the g-th neural network to be evaluated based on the N initial accuracy score values.

[0083] According to an embodiment of the present invention, the accuracy score can be obtained by using several small batches of training data to obtain gradients, and comparing the consistency of the gradients obtained in different batches to represent the oscillation probability of the network weight update. Specifically, the consistency within each network layer of the neural network to be evaluated is summed, the consistency between different network layers is multiplied, and the logarithmic function is used to scale to avoid the accuracy score value being too large. For example, the neural network to be evaluated includes 3 network layers. For the first network layer, there are 4 weight parameters corresponding to the first network layer. The ratio between the gradient mean and the gradient variance corresponding to the first weight parameter is determined to obtain the initial accuracy score value of the first weight parameter corresponding to the first network layer, and then the initial accuracy score value of each of the 4 weight parameters corresponding to the first network layer is obtained; after summing the initial accuracy score values ​​of each of the 4 weight parameters corresponding to the first network layer, the initial accuracy score value of the first network layer is obtained, and then the initial accuracy score value of each of the 3 network layers is obtained. The initial accuracy score values ​​of each of the 3 network layers are multiplied to determine the accuracy score value of the gth neural network to be evaluated.

[0084] Specifically, the accuracy score value can be determined as shown in formula (2).

[0085] (2);

[0086] Where D represents the number of network layers, Represents all weight parameters of the d-th network layer, represents the target gradient interval, represents the indicator function, represents the gradient average of the nth target gradient value set, Represents the gradient variance value of the nth target gradient value set, represents the accuracy score of the gth neural network to be evaluated. In addition, the sign of the gradient value is retained before taking the absolute value to capture the direction of the gradient update and the oscillation of the weight. log represents the logarithm.

[0087] According to an embodiment of the present invention, a higher precision score indicates a more stable gradient across different batches, and the corresponding training loss of the network architecture of the neural network to be evaluated decreases faster. Because the precision score determination process of the present invention eliminates the training process used in the related art for calculating network rankings, it avoids the need to design a cumbersome and complex supernet, reduces the computational overhead of service devices such as computers, and improves the search accuracy of the target neural network while ensuring an increase in the computational speed of service devices such as computers, thereby reducing the burden on service devices such as computers.

[0088] According to an embodiment of the present invention, a heterogeneous multi-core architecture includes multiple hardware cores; hardware performance simulation is performed on the hardware configuration parameters of the neural network to be evaluated using the heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated, including: mapping the g-th neural network to be evaluated to multiple hardware cores to obtain multiple target hardware cores; determining the target number of computing units and target bandwidth of each target hardware core based on the ratio of the computing demand and the bandwidth demand of each target hardware core; optimizing the data flow parameters of each target hardware core to obtain the target data flow parameters of each target hardware core; using a performance simulation framework, hardware performance simulation is performed on the target number of computing units, target bandwidth and target data flow parameters of each target hardware core to determine the hardware performance value of the g-th neural network to be evaluated.

[0089] Figure 3 A schematic diagram of a heterogeneous multi-core architecture according to an embodiment of the present invention is schematically shown.

[0090] According to an embodiment of the present invention, Figure 3 As shown in FIG, a heterogeneous multi-core architecture can be a multi-core processor architecture with multiple hardware cores. The computing sub-unit (PE) allocation, data flow scheduling, etc. of each hardware core are different. Among them, the data and performance calculation results required by the computing sub-units interact with the cache (global cache, local cache) through multi-level interconnection networks (global interconnection, local interconnection).

[0091] According to an embodiment of the present invention, the neural network to be evaluated can be decomposed into a series of Q operators to be executed The heterogeneous multi-core architecture is denoted as , represents a heterogeneous multi-core architecture with H hardware cores.

[0092] According to an embodiment of the present invention, the neural network to be evaluated is mapped to multiple hardware cores, which can be recorded as , then the mapping of operators to cores can be expressed as an assignment problem. That is, we need a mapping , satisfying the set arrive Specifically, a heterogeneous multi-core architecture can be used to simulate a hardware accelerator, and the neural network to be evaluated can be mapped to multiple hardware cores of the heterogeneous multi-core architecture as the load of the hardware accelerator.

[0093] According to an embodiment of the present invention, the ratio of the computing demand and the bandwidth demand of each target hardware core can be determined based on the preset hardware constraint requirements, so as to allocate the computing demand and the bandwidth demand of each target hardware core to obtain the target number of computing units and target bandwidth of each target hardware core.

[0094] According to an embodiment of the present invention, dataflow parameters describe the scheduling of operators between the target hardware core's compute units and memory, further reflecting the interconnection structure between the compute units. Specifically, they can include the six-dimensional loop order, parallel dimension, slice size, and degree of parallelism for each target hardware core's operator in a one-dimensional array. These parameters are repeated at the next level in the two-dimensional array, resulting in an exponential growth in the dataflow parameter space of the target hardware core.

[0095] According to an embodiment of the present invention, the data flow parameters of each target hardware core are optimized based on a genetic algorithm to select the optimal data flow configuration for the operator of the target hardware core to obtain the target data flow parameters of each target hardware core.

[0096] According to an embodiment of the present invention, each target hardware core can be represented as , represented by the target data flow parameters, target number of computing units, and target bandwidth of the hth target hardware core. Once the target hardware cores are determined, these data flow parameters, target number of computing units, and target bandwidth are fed into a mature performance simulation framework, such as Maestro or Timegoop. This yields the hardware performance values ​​of the neural network to be evaluated, representing the power consumption, area, and latency of the target hardware core.

[0097] According to an embodiment of the present invention, the operation of determining the hardware performance value of the present invention covers data scheduling, data parallelism and hardware interconnection structure, and realizes adaptation to different neural networks and different operators. Under similar hardware performance evaluation time, the hardware performance output by the heterogeneous multi-core architecture is improved under different applications, and the improvement of the energy consumption delay product can be up to 235.8 times, thereby stabilizing the operation of service equipment such as computers and reducing the computing time of service equipment such as computers.

[0098] According to an embodiment of the present invention, the g-th neural network to be evaluated includes multiple operators; mapping the g-th neural network to be evaluated to multiple hardware cores to obtain multiple target hardware cores, including: clustering all operators according to the dimension parameters of each operator to obtain the category of each operator; mapping operators with the same category to the same hardware core to obtain multiple target hardware cores.

[0099] According to an embodiment of the present invention, the dimensional parameters of each operator include six dimensional parameters, using K, C, R, S, X, and Y to represent the number of output channels, the number of input channels, the height of the convolution kernel, the width of the convolution kernel, and the height and width of the input feature map, respectively. Among them, the matrix multiplication operator is considered to be equivalent to a convolution calculation with a convolution kernel size of 1, thus covering the main operators of convolutional neural networks and Transformer architecture-type neural network workloads.

[0100] According to an embodiment of the present invention, mapping only needs to care about the similarity of the computing characteristics between each operator, and uniformly map similar operators to the same hardware core. Specifically, the DB-scan clustering algorithm is adopted, based on the above six dimensional parameters, and performs operator clustering in a high-dimensional space, wherein similar values ​​on different dimensional parameters mean similar preferences on data reuse patterns and parallel dimensions. The above clustering algorithm classifies operators by the distance between adjacent points, and can automatically determine the number of operator categories based on the situation of the clustering points, wherein the similarity distance between different operators is calculated using Euclidean distance.

[0101] According to an embodiment of the present invention, a heterogeneous multi-core architecture can design a common parallel solution for operators with similar computing characteristics, which is more adaptable than the existing technology in which all operators share the same set of hardware performance calculation methods. Therefore, there is no need to build different architectures for different operators, which can reduce the storage space occupied by service devices such as computers.

[0102] According to an embodiment of the present invention, the data flow parameters of each target hardware core are optimized to obtain the target data flow parameters of each target hardware core, including: for the data flow parameters of any target hardware core, the data flow parameters are iteratively optimized to obtain candidate data flow parameters; and the candidate data flow parameters that meet the preset iteration conditions of the data flow parameters are determined as the target data flow parameters.

[0103] According to an embodiment of the present invention, the data stream parameters are iteratively optimized based on a genetic algorithm, and the data stream parameters of each iteration serve as individuals of each generation in the genetic algorithm, ie, candidate data stream parameters.

[0104] Figure 4 The figure schematically shows data flow parameters according to an embodiment of the present invention.

[0105] According to an embodiment of the present invention, taking one-dimensional data stream parameters as an example, the data stream parameters include the order of different dimensions, parallel dimension, parallelism, and shard size, which can be represented by a 2x7 genome. Figure 4 As shown, based on the load description of the neural network to be evaluated corresponding to each operator of the neural network to be evaluated, the operators are mapped to the hardware cores, and the target hardware cores are encoded to obtain the data flow parameters, where the first line is the parallel dimension (corresponding to Figure 4 The first column "K" and the eighth column "C") and the calculation order (corresponding to Figure 4The second to seventh columns ("SY" and the ninth to fourteenth columns ("SR") correspond to the computation order of the six dimensional parameters of the corresponding operators. The second row indicates the corresponding degree of parallelism and shard size. Furthermore, a two-dimensional array can be viewed as a superposition of multiple one-dimensional arrays, so the encoding can be represented using a 2×14 genome. In this case, columns 8–14 represent the one-dimensional arrays, and columns 1–7 indicate the scheduling and parallelization within that one dimension.

[0106] According to an embodiment of the present invention, based on the characteristics of a genetic algorithm, candidate data flow parameters with high data flow parameter fitness are selected to generate new candidate data flow parameters. Specifically, new candidate data flow parameters are generated from existing candidate data flow parameters. During the optimization process, mutations are performed on individual candidate data flow parameters through crossover mutation updates. These can include swapping loop order, modifying parallel dimensions, or mutating parallelism and sharding size. New data flow parameters are generated for multiple data flow parameters in each generation. These can also involve crossover of two-dimensional and one-dimensional genes, crossover of parallel dimensions, and so on.

[0107] According to an embodiment of the present invention, the iterative convergence period of the genetic algorithm exhibits a degree of randomness, varying depending on the crossover and mutation probability and the operator. To ensure the quality of iterative optimization, the preset iteration conditions may include setting the number of iterations and selecting the optimal candidate data stream parameters as the target data stream parameters. For situations where iterative convergence is rapid, an early stopping mechanism is employed to reduce search time. If the candidate data stream parameters remain unupdated for three consecutive iterations, convergence is reached and the iterative optimization is terminated prematurely.

[0108] According to an embodiment of the present invention, before optimizing the data flow parameters of each target hardware core, the method further includes: pruning the data flow parameters of any target hardware core to obtain pruned data flow parameters.

[0109] According to an embodiment of the present invention, to reduce the computational burden on hardware performance, data stream parameters are pruned based on both value and policy. Value-based pruning constrains the explored values ​​for slice size and parallelism to multiples of 4 during two-dimensional array exploration. This significantly reduces hardware performance search points and aligns with hardware design experience. Policy-based pruning selects representative operators for data stream search based on the load borne by the hardware core.

[0110] According to an embodiment of the present invention, when pruning is not performed, the data flow parameters of a hardware core must be used to calculate the hardware performance of all operators. This repetitive process is time-consuming, and the evaluation time of the computer system backend evaluation tool increases linearly with the number of operators. In addition, because operators mapped to the same hardware core are clustered during the mapping phase, performing hardware performance calculations on all operators is redundant. Policy-based pruning searches for the optimal data flow parameter configuration for each operator and applies this data flow parameter to all operators on the hardware core.

[0111] According to the embodiment of the present invention, data flow parameters of any target hardware core are pruned, thereby avoiding data redundancy and improving the computing speed of service equipment such as computers.

[0112] Figure 5 A schematic diagram of a joint search architecture of a neural network and hardware according to an embodiment of the present invention is schematically shown.

[0113] According to an embodiment of the present invention, the above-mentioned neural network and hardware joint search method of the present invention can be implemented as follows: Figure 5 The neural network joint search architecture is implemented as shown, wherein the neural network joint search architecture includes a neural network search space, an accuracy evaluator and a hardware searcher.

[0114] Preset neural network search conditions, training label data, and hardware constraint requirements are input into the neural network joint search architecture. The neural network search space corresponding to the current iteration is sampled based on the preset neural network search conditions, and a neural network population including G neural networks to be evaluated is output. The accuracy evaluator determines a target gradient value set for the gth neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration, and determines the accuracy score value of the gth neural network to be evaluated based on the target gradient value set of the gth neural network to be evaluated. The hardware searcher performs hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated using a heterogeneous multi-core architecture to determine the hardware performance value of the gth neural network to be evaluated. The neural network joint search architecture then determines the fitness value of the gth neural network to be evaluated based on the ratio between the accuracy score value and the hardware performance value of the gth neural network to be evaluated. The neural network to be evaluated with the highest fitness value among the obtained G neural networks to be evaluated is determined as the candidate neural network corresponding to the current iteration. Based on the candidate neural networks corresponding to the current iteration, the sampling distribution of the neural network search space for the next iteration is optimized, and the candidate neural network that meets the predetermined iteration conditions is determined as the target neural network and output.

[0115] According to an embodiment of the present invention, by combining the designs of the aforementioned accuracy estimator and hardware searcher, the present invention achieves both speed and efficiency in joint search scenarios. Specifically, a neural network joint search architecture was evaluated for convolutional neural networks and visual transformers based on the ImageNet (ImageNet Large Scale Visual Recognition Challenge) and CIFAR-10 (Canadian Institute for Advanced Research) datasets. Experimental results demonstrate that the neural network joint search architecture of the present invention efficiently explores the neural network search space. Compared with existing search methods, the present invention saves 12.96 times the energy-delay product and improves search efficiency by 48 times. Furthermore, compared to existing technologies, the neural network searched by the present invention achieves a 56% energy-delay product reduction while maintaining similar network accuracy, reducing search time from 420 hours to approximately 3 hours.

[0116] According to an embodiment of the present invention, the following test operations were performed on the aforementioned accuracy evaluator. The accuracy evaluator ranked the accuracy scores of neural networks for both convolutional neural network and visual transformer scenarios, and verified the reliability of the rankings based on datasets of various sizes. As shown in Table 1, the CNN (Convolutional Neural Network) benchmark (NASBench-201) was derived from the invented datasets (CIFAR10, CIFAR100, and ImgNet16), while the ViT (Vision Transformer) benchmark (AutoFormer) was derived from a sampling of 1,000 random neural networks (ImageNet-1k) from an existing supernet, and their actual test accuracy was obtained. The reliability evaluation metrics used were the Spelman coefficient (S) and the Kendall coefficient (K), verifying the consistency between the predicted accuracy rankings and the actual accuracy rankings generated by the accuracy evaluator of the present invention. Both coefficients are in the range [-1, 1]. A larger coefficient indicates a higher correlation, i.e., a more reliable ranking.

[0117]

[0118] Results show that the prediction accuracy ranking and actual accuracy provided by the accuracy estimator of the present invention maintain good consistency compared to those of related art estimators (ZenScore, ZiCo, GradSign, #Param, #FLOPs, Snip, DSS, Param, FLOPs). For example, the Spelman coefficient (S) and Kendall coefficient (K) measured by the present invention reached approximately 0.64 and 0.82, respectively, demonstrating the reliability of accuracy assessment using the accuracy estimator of the present invention. Furthermore, the accuracy estimator of the present invention maintains the highest consistency across neural network search spaces and across datasets, providing a guarantee for model comparison during search.

[0119] In addition, the neural network search results after using the accuracy evaluator of the present invention can be shown in Table 2.

[0120]

[0121] It can be seen that compared to evaluators in related technologies, the neural networks searched by the accuracy evaluator designed by the present invention exhibit the highest neural network accuracy scores across various data sets. For example, in the visual transformer scenario, the accuracy evaluator of the present invention yielded a higher neural network accuracy score. Furthermore, because the accuracy evaluator of the present invention is training-free, the search time can be reduced to 3 hours compared to other evaluators, resulting in a 480x improvement in search efficiency. This, in turn, reduces the computing time of service devices such as computers while also reducing the storage space occupied by them.

[0122] According to an embodiment of the present invention, the following test operations were performed on the aforementioned hardware searcher. Classic network models under CNN and ViT were selected, covering both large-scale and small-scale models. The hardware searcher of the present invention was applied to perform hardware performance searches. The hardware constraint for small-scale networks (ResNet18, EfficientNetB1, ViT-Tiny, PiT-Tiny) was 96 computing units, while the hardware constraint for large-scale networks (ResNet50, VGG16, ViT-Small, PiT-Small) was 1024 hardware units. The goal of the hardware performance search was to minimize the energy-delay product (EDP). Single-core hardware search, naive operator partitioning, and unpruned search were used as the comparison benchmarks. The search results and search times under various strategies (single-core, staged, pruned, and the present invention) were compared.

[0123] Figure 6 The figure schematically shows the test results of the hardware searcher according to an embodiment of the present invention.

[0124] like Figure 6 As shown in the figure, the horizontal axis represents ResNet18, ResNet50, EfficientNetB1, Convolutional Neural Network (VGG16), Visual Transformer-Tiny, Visual Transformer-Small, Pyramid Visual Transformer-Tiny, and Pyramid Visual Transformer-Small. The left side of the vertical axis represents 1E+11, 1E+12, ..., representing different energy-delay products, and the right side represents search time. The figure shows the EDP and search time under different search strategies (single-core, stage-based, without purging, and the hardware searcher of the present invention) in the aforementioned small-scale and large-scale network environments, namely the four columns and broken lines corresponding to each network environment. The results show that the hardware searcher of the present invention significantly improves EDP, with improvements ranging from 4.3x to 192.6x in CNN and 2.7x to 235.8x in ViT. This is due to the parallel adaptation of heterogeneous multi-core architectures to heterogeneous operators. Small-scale networks achieve significantly better gains than large-scale networks, demonstrating the necessity of heterogeneous multi-cores in resource-constrained scenarios. Furthermore, search space pruning significantly reduces search time, improving search efficiency by 2.2x to 7.4x compared to the unpruned version while maintaining similar search results. This demonstrates that the proposed pruning strategy preserves the optimal solution space while effectively removing redundant search points.

[0125] According to an embodiment of the present invention, the following test operations were performed on the aforementioned neural network joint search architecture. A joint search was conducted for both CNN and ViT scenarios, focusing on hardware adaptation and performance while pursuing neural network accuracy. This invention uses a training-free evaluator and hardware searcher to search for high-performing neural networks, then trains a single neural network for application. The search results are shown in Table 3.

[0126] In a CNN scenario, this paper uses the neural network search space of ProxylessNAS to perform neural network exploration. Compared to existing methods (DANCE and DIAN), the search results of this paper demonstrate high accuracy and high hardware efficiency (EDP). The neural network achieved 73.96% accuracy, an improvement of 1.76% to 3.81%, while also improving the energy-delay product by 3.54x and 12.96x, respectively. Furthermore, this paper eliminates the need for pre-training and data collection, and the search time on server devices such as computers is only 3 hours, achieving a 48x efficiency improvement.

[0127] For ViT scenarios, compared with existing methods (AutoF-T, AutoF-S, AutoF-B), the neural network joint search architecture of the present invention maintains similar neural network accuracy performance, with a maximum accuracy loss of 0.2%. However, in terms of hardware efficiency, the search results of the present invention are improved by 14% to 36%. At the same time, the search time is increased by 131 to 161 times, from 420 hours to about 3 hours, thereby reducing the search time of service equipment such as computers.

[0128]

[0129] Based on the above-mentioned neural network and hardware joint search method, the present invention also provides a neural network joint search device. Figure 7 The device is described in detail.

[0130] Figure 7 The structural block diagram of the neural network joint search device according to an embodiment of the present invention is schematically shown.

[0131] like Figure 7 As shown, the neural network joint search device 700 of this embodiment includes a storage unit 710, a computing unit 720 and a heterogeneous multi-core architecture 730.

[0132] The storage unit 710 is used to store the neural network search space.

[0133] The calculation unit 720 is configured to: based on preset neural network search conditions, sample the neural network search space corresponding to the current iteration in the storage unit 710 to obtain G neural networks to be evaluated corresponding to the current iteration; perform a gradient calculation task on the gth neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration to obtain a target gradient value set for the gth neural network to be evaluated, where G ≥ 1 and 1 ≤ g ≤ G; and determine the accuracy score of the gth neural network to be evaluated based on the target gradient value set for the gth neural network to be evaluated.

[0134] The heterogeneous multi-core architecture 730 is configured to perform hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated, and determine the hardware performance value of the g-th neural network to be evaluated.

[0135] The computing unit 720 is further configured to: determine the adaptability value of the g-th neural network to be evaluated based on the ratio between the accuracy score value and the hardware performance value of the g-th neural network to be evaluated, and obtain the adaptability values ​​of G neural networks to be evaluated; determine the neural network to be evaluated corresponding to the highest adaptability value among the adaptability values ​​of the G neural networks to be evaluated as the candidate neural network corresponding to the current round of iteration; and determine the target neural network suitable for executing the target processing task from the candidate neural networks corresponding to each of the multiple rounds.

[0136] According to an embodiment of the present invention, based on preset neural network search conditions, the neural network search space corresponding to the current round iteration in the storage unit 710 is sampled to obtain G neural networks to be evaluated corresponding to the current round iteration; based on the g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current round iteration, the computing unit 720 is used to perform a gradient calculation task to obtain a target gradient value set for the g-th neural network to be evaluated; based on the target gradient value set of the g-th neural network to be evaluated, the accuracy score value of the g-th neural network to be evaluated is determined; hardware performance simulation is performed on the hardware configuration parameters of the neural network to be evaluated using the heterogeneous multi-core architecture 730 to determine the hardware performance value of the g-th neural network to be evaluated; based on the ratio between the accuracy score value and the hardware performance value of the g-th neural network to be evaluated, the adaptability value of the g-th neural network to be evaluated is determined to obtain the adaptability values ​​of the G neural networks to be evaluated; the neural network to be evaluated corresponding to the highest adaptability value among the G neural networks to be evaluated is determined as the candidate neural network corresponding to the current round iteration; and a target neural network suitable for performing the target processing task is determined from the candidate neural networks corresponding to each of the multiple rounds. Since the computing unit 720 is used to perform the gradient calculation task, the relationship between the gradient information and the model convergence of the neural network to be evaluated is fully utilized when determining the accuracy score value corresponding to the neural network to be evaluated, thereby avoiding the design of a heavy and complex supernet, reducing the computing overhead of service equipment such as computers, and improving the search accuracy of the target neural network while ensuring the computing speed of service equipment such as computers. In addition, since the hardware performance of the neural network to be evaluated is simulated by the heterogeneous multi-core architecture 730, the problem of the single-core architecture failing to fully consider the differences and computing preferences of the network layers is avoided, so that service equipment such as computers does not need to reconfigure the hardware performance evaluation architecture for different neural networks to be evaluated, thereby ensuring the stable operation of service equipment such as computers and reducing the storage space occupied by service equipment such as computers.

[0137] According to embodiments of the present invention, any multiple modules in the storage unit 710, computing unit 720, and heterogeneous multi-core architecture 730 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present invention, at least one of the storage unit 710, computing unit 720, and heterogeneous multi-core architecture 730 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the storage unit 710, computing unit 720, and heterogeneous multi-core architecture 730 can be at least partially implemented as a computer program module that, when executed, can perform the corresponding functionality.

[0138] According to an embodiment of the present invention, the calculation unit 720 is further configured to:

[0139] Determine M batches of training label data to be input into the g-th neural network to be evaluated, where M ≥ 1.

[0140] For the mth batch of training label data in the M batches, determine the loss value of the mth batch of the gth neural network to be evaluated according to the mth batch of training label data of the gth neural network to be evaluated, 1≤m≤M.

[0141] Calculate the loss value of the mth batch and the derivatives of the weight parameters of the N gth neural networks to be evaluated respectively, obtain N gradient values ​​of the mth batch, and obtain M×N gradient values ​​of M batches, where N≥1.

[0142] The M×N gradient values ​​are screened to obtain the target gradient value set of the g-th neural network to be evaluated.

[0143] According to an embodiment of the present invention, the calculation unit 720 is further configured to:

[0144] For the nth weight parameter among the N weight parameters of the gth neural network to be evaluated, the M gradient values ​​corresponding to the nth weight parameter are divided into intervals to obtain multiple gradient intervals, 1≤n≤N.

[0145] According to the density of each gradient interval, the gradient interval with the largest density is selected from multiple gradient intervals as the nth target gradient interval corresponding to the nth weight parameter, thereby obtaining N target gradient intervals.

[0146] According to the nth target gradient interval, the nth target gradient value set corresponding to the nth weight parameter is determined to obtain N target gradient value sets.

[0147] The N target gradient value sets are used as the target gradient value sets of the g-th neural network to be evaluated.

[0148] According to an embodiment of the present invention, the calculation unit 720 is further configured to:

[0149] For the nth weight parameter of the gth neural network to be evaluated, determine the gradient average of the nth target gradient value set.

[0150] Determine the gradient variance value of the nth target gradient value set.

[0151] The nth initial accuracy score value is determined according to the ratio between the gradient mean value and the gradient variance value, thereby obtaining N initial accuracy score values.

[0152] Based on the N initial accuracy scores, determine the accuracy score of the g-th neural network to be evaluated.

[0153] According to an embodiment of the present invention, the heterogeneous multi-core architecture 730 is further configured to:

[0154] Map the g-th neural network to be evaluated to multiple hardware cores to obtain multiple target hardware cores.

[0155] The target number of computing units and target bandwidth of each target hardware core are determined based on the ratio of the computing demand and bandwidth demand of each target hardware core.

[0156] The data flow parameters of each target hardware core are optimized to obtain target data flow parameters of each target hardware core.

[0157] Using the performance simulation framework, hardware performance simulation is performed on the target number of computing units, target bandwidth, and target data flow parameters of each target hardware core to determine the hardware performance value of the g-th neural network to be evaluated.

[0158] According to an embodiment of the present invention, the heterogeneous multi-core architecture 730 is further configured to:

[0159] According to the dimension parameters of each operator, all operators are clustered to obtain the category of each operator.

[0160] Operators of the same category are mapped to the same hardware core to obtain multiple target hardware cores.

[0161] According to an embodiment of the present invention, the heterogeneous multi-core architecture 730 is further configured to:

[0162] For the data flow parameters of any target hardware core, the data flow parameters are iteratively optimized to obtain candidate data flow parameters.

[0163] The candidate data flow parameters that meet the preset iteration conditions of the data flow parameters are determined as the data flow parameters.

[0164] According to an embodiment of the present invention, the heterogeneous multi-core architecture 730 is further configured to:

[0165] Prune the data flow parameters of any target hardware core to obtain pruned data flow parameters.

[0166] Figure 8 The figure schematically shows a block diagram of an electronic device suitable for implementing a joint search method of a neural network and hardware according to an embodiment of the present invention.

[0167] like Figure 8 As shown, an electronic device according to an embodiment of the present invention includes a processor 801, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 802 or programs loaded from a storage unit 808 into a random access memory (RAM) 803. Processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets, and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)). Processor 801 may also include onboard memory for caching purposes. Processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0168] Various programs and data required for the operation of the electronic device are stored in RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the programs in ROM 802 and / or RAM 803 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. The processor 801 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0169] According to an embodiment of the present invention, the electronic device may further include an input / output (I / O) interface 805, which is also connected to the bus 804. Electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. Removable media 811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in the drive 810 as needed, so that computer programs read from the removable media can be installed into the storage section 808 as needed.

[0170] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0171] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above, and / or one or more memories other than ROM 802 and RAM 803.

[0172] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is configured to cause the computer system to implement the neural network and hardware joint search method provided in an embodiment of the present invention.

[0173] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when executed by the processor 801. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0174] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0175] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809 and / or installed from a removable medium 811. When the computer program is executed by the processor 801, the above-described functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0176] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0179] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A joint search method of neural network and hardware, characterized in that: Applied to a computing device, the method includes: Based on the preset neural network search conditions, the neural network search space corresponding to the current iteration in the storage unit is sampled to obtain G neural networks to be evaluated corresponding to the current iteration; Based on the g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration, using a computing unit to perform a gradient calculation task to obtain a target gradient value set of the g-th neural network to be evaluated, G≥1, 1≤g≤G; Determining an accuracy score value of the g-th neural network to be evaluated based on the target gradient value set of the g-th neural network to be evaluated; Performing hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated using multiple target hardware cores among the multiple hardware cores included in the heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated, wherein the g-th neural network to be evaluated includes multiple operators, and clustering all the operators according to the dimension parameters of each operator to obtain a category of each operator; mapping the operators with the same category to the same hardware core to obtain multiple target hardware cores; Determining the adaptability value of the g-th neural network to be evaluated according to the ratio between the accuracy score value of the g-th neural network to be evaluated and the hardware performance value, thereby obtaining the adaptability values ​​of the G neural networks to be evaluated; Determine the neural network to be evaluated corresponding to the highest fitness value among the G neural networks to be evaluated as the candidate neural network corresponding to the current iteration; A target neural network suitable for performing the target processing task is determined from the candidate neural networks corresponding to multiple rounds.

2. The method according to claim 1, characterized in that The target processing task includes at least one of the following: Object detection tasks, text processing tasks, and image classification tasks.

3. The method according to claim 1, characterized in that The step of performing a gradient calculation task using a computing unit based on a g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration to obtain a target gradient value set for the g-th neural network to be evaluated includes: Determine M batches of training label data to be input into the g-th neural network to be evaluated, where M ≥ 1; For the mth batch of training label data in the M batches, determine the loss value of the mth batch of the gth neural network to be evaluated according to the mth batch of training label data of the gth neural network to be evaluated, where 1≤m≤M; Calculate the loss value of the mth batch and the derivatives of the N weight parameters of the gth neural network to be evaluated respectively, obtain N gradient values ​​of the mth batch, and obtain M×N gradient values ​​of the M batches, where N≥1; The M×N gradient values ​​are screened to obtain a target gradient value set for the g-th neural network to be evaluated.

4. The method according to claim 3, characterized in that The screening of the M×N gradient values ​​to obtain the target gradient value set of the g-th neural network to be evaluated includes: For an nth weight parameter among the N weight parameters of the gth neural network to be evaluated, dividing the M gradient values ​​corresponding to the nth weight parameter into intervals to obtain a plurality of gradient intervals, where 1≤n≤N; According to the density of each gradient interval, the gradient interval with the largest density is selected from the plurality of gradient intervals as the nth target gradient interval corresponding to the nth weight parameter, to obtain N target gradient intervals; Determining an nth target gradient value set corresponding to the nth weight parameter according to the nth target gradient interval, to obtain N target gradient value sets; The N target gradient value sets are used as the target gradient value sets of the g-th neural network to be evaluated.

5. The method according to claim 4, characterized in that Determining the accuracy score value of the g-th neural network to be evaluated based on the target gradient value set of the g-th neural network to be evaluated includes: For the nth weight parameter of the gth neural network to be evaluated, determining the gradient average of the nth target gradient value set; Determining the gradient variance value of the nth target gradient value set; Determine an nth initial accuracy score value according to a ratio between the gradient mean value and the gradient variance value, to obtain N initial accuracy score values; Determine the accuracy score value of the g-th neural network to be evaluated based on the N initial accuracy score values.

6. The method according to claim 1, characterized in that The performing hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated using multiple target hardware cores among the multiple hardware cores included in the heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated includes: Determining the target number of computing units and the target bandwidth of each target hardware core according to the ratio of the computing demand and the bandwidth demand of each target hardware core; Optimizing the data flow parameters of each target hardware core to obtain target data flow parameters of each target hardware core; Using a performance simulation framework, hardware performance simulation is performed on the target number of computing units, the target bandwidth, and the target data flow parameters of each target hardware core to determine the hardware performance value of the g-th neural network to be evaluated.

7. The method according to claim 6, characterized in that Optimizing the data flow parameters of each target hardware core to obtain target data flow parameters of each target hardware core includes: For any data flow parameter of the target hardware core, iteratively optimize the data flow parameter to obtain candidate data flow parameters; The candidate data stream parameters that meet the preset iteration condition of the data stream parameters are determined as the data stream parameters.

8. The method according to claim 6, characterized in that Before optimizing the data flow parameters of each target hardware core, the method further includes: Pruning is performed on the data flow parameters of any of the target hardware cores to obtain pruned data flow parameters.

9. A joint search device of neural network and hardware, characterized in that: include: A storage unit for storing the neural network search space; Computing unit, configured as: Based on the preset neural network search conditions, sampling the neural network search space corresponding to the current round iteration in the storage unit to obtain G neural networks to be evaluated corresponding to the current round iteration; Performing a gradient calculation task based on a g-th neural network to be evaluated among the G neural networks to be evaluated corresponding to the current iteration to obtain a target gradient value set for the g-th neural network to be evaluated, G ≥ 1, 1 ≤ g ≤ G; Determining an accuracy score value of the g-th neural network to be evaluated based on the target gradient value set of the g-th neural network to be evaluated; Heterogeneous multi-core architecture, configured as: Performing hardware performance simulation on the hardware configuration parameters of the neural network to be evaluated using multiple target hardware cores among the multiple hardware cores included in the heterogeneous multi-core architecture to determine the hardware performance value of the g-th neural network to be evaluated, wherein the g-th neural network to be evaluated includes multiple operators, and clustering all the operators according to the dimension parameters of each operator to obtain a category of each operator; mapping the operators with the same category to the same hardware core to obtain multiple target hardware cores; The computing unit is further configured to: determine the adaptability value of the g-th neural network to be evaluated based on the ratio between the accuracy score value of the g-th neural network to be evaluated and the hardware performance value, and obtain the adaptability values ​​of the G neural networks to be evaluated; determine the neural network to be evaluated corresponding to the highest adaptability value among the adaptability values ​​of the G neural networks to be evaluated as the candidate neural network corresponding to the current round of iteration; and determine a target neural network suitable for executing the target processing task from the candidate neural networks corresponding to each of the multiple rounds.

Images