Hardware system model generation method and device, equipment, medium and program product

Abstract

The invention relates to a hardware system model generation method and device, equipment, a medium and a program product. The method comprises the following steps: carrying out data sampling based on a distribution condition of a nonlinear activation function in a visual deformer to obtain visual distribution nonlinear data; performing segmented optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data; performing replacement fine tuning on a pre-trained original visual model based on the visual distribution linear model to obtain a target visual model; generating a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used for controlling the visual deformer. By adopting the method, the hardware implementation process of the activation function can be optimized, the function calculation complexity and the hardware implementation cost and efficiency are reduced, and meanwhile, the reasoning precision of a hardware model can be ensured to a great extent.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method, device, equipment, medium and program product for generating a hardware system model. Background Art

[0002] With the development of deep learning technology, activation functions are often used in deep learning to build hardware models. However, in the specific hardware implementation process, complex and inefficient operators (such as exponential calculations, power operations, etc.) are usually used, which will lead to a large performance bottleneck in the end-side reasoning of deep learning models. Faced with the hardware implementation of highly complex activation functions, there is an urgent need for a hardware system model generation method that can optimize the hardware implementation process of the activation function, reduce the complexity of function calculations and the cost and efficiency of hardware implementation, and at the same time guarantee the accuracy of hardware model reasoning to a great extent. Summary of the invention

[0003] Based on this, it is necessary to provide a hardware system model generation method, device, computer equipment, computer-readable storage medium and computer program product that can optimize the hardware implementation process of the activation function, reduce the function calculation complexity and the cost and efficiency of hardware implementation, and at the same time ensure the hardware model reasoning accuracy to the greatest extent, in response to the above-mentioned technical problems.

[0004] In a first aspect, the present application provides a method for generating a hardware system model, comprising:

[0005] Data sampling is performed based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data;

[0006] Performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data;

[0007] Based on the visual distribution linear model, the pre-trained original visual model is replaced and fine-tuned to obtain a target visual model;

[0008] A hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0009] In one embodiment, the step of performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data includes:

[0010] Get hyperparameter data;

[0011] Performing segmented optimization setting on the visual distribution nonlinear data according to the hyperparameter data to obtain visual distribution linear optimization data;

[0012] The visual distribution linear optimization data is evolutionarily updated based on the evolution model to obtain the visual distribution linear data that meets preset conditions.

[0013] In one embodiment, constructing a visual distribution linear model according to the visual distribution linear data includes:

[0014] Performing derivable activation processing according to the visual distribution linear data to obtain replaceable linear model data; wherein the replaceable linear model data is used for performing derivation processing in an unrestricted direction;

[0015] The visual distribution linear model is constructed based on the replaceable linear model data.

[0016] In one embodiment, replacing and fine-tuning the pre-trained original visual model based on the visual distribution linear model to obtain the target visual model includes:

[0017] Replacing the nonlinear sub-model of the original visual model with the visual distribution linear model to obtain a visual model to be fine-tuned;

[0018] Quantization perception fine-tuning is performed based on the visual model to be fine-tuned to obtain the target visual model.

[0019] In one embodiment, generating a hardware system based on the target visual model to obtain a target hardware system model includes:

[0020] Replacing the target linear sub-model format of the target visual model with the original nonlinear model format of the original visual model to obtain a target visual nonlinear model;

[0021] A hardware system is generated based on the quantized parameters of the target visual nonlinear model to obtain the target hardware system model.

[0022] In one embodiment, the method further comprises:

[0023] Performing performance testing based on the target hardware system model to obtain performance testing data;

[0024] A performance evaluation is performed on the target visual linear model of the target hardware system model according to the performance test data to obtain performance evaluation data.

[0025] In a second aspect, the present application also provides a hardware system model generation device, including:

[0026] A data sampling module, used for performing data sampling based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data;

[0027] A segmented optimization module, used for segmentally optimizing the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data;

[0028] A replacement fine-tuning module, used for performing replacement fine-tuning on the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model;

[0029] The hardware system generation module is used to generate a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0031] Data sampling is performed based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data;

[0032] Performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data;

[0033] Based on the visual distribution linear model, the pre-trained original visual model is replaced and fine-tuned to obtain a target visual model;

[0034] A hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0036] Data sampling is performed based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data;

[0037] Performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data;

[0038] Based on the visual distribution linear model, the pre-trained original visual model is replaced and fine-tuned to obtain a target visual model;

[0039] A hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0040] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0041] Data sampling is performed based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data;

[0042] Performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data;

[0043] Based on the visual distribution linear model, the pre-trained original visual model is replaced and fine-tuned to obtain a target visual model;

[0044] A hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0045] The above-mentioned hardware system model generation method, device, computer equipment, computer-readable storage medium and computer program product obtain visual distribution nonlinear data by sampling data based on the distribution of nonlinear activation functions in the visual deformer; optimize the visual distribution nonlinear data in segments to obtain visual distribution linear data, and construct a visual distribution linear model based on the visual distribution linear data; replace and fine-tune the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model; generate a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer. Therefore, by performing segmented optimization on the visual distribution nonlinear data, linear data that is easy to calculate, namely, visual distribution linear data, is obtained, and then the hardware implementation process of the activation function is further optimized by replacement and fine-tuning processing, reducing the function calculation complexity and the cost and efficiency of hardware implementation. At the same time, the hardware system is generated based on the target visual model generated after fine-tuning, and the obtained target hardware system model can also guarantee the reasoning accuracy of the target hardware system model to a great extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] FIG1 is an application environment diagram of a method for generating a hardware system model in one embodiment;

[0048] FIG2 is a schematic diagram of a process flow of a method for generating a hardware system model in one embodiment;

[0049] FIG3 is a schematic diagram of a flow chart of a method for generating a hardware system model in one embodiment;

[0050] FIG4 is a schematic flow chart of a method for generating a hardware system model in another embodiment;

[0051] FIG5 is a structural block diagram of a hardware system model generating device in one embodiment;

[0052] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The hardware system model generation method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1. The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 performs data sampling based on the distribution of the nonlinear activation function in the visual deformer to obtain visual distribution nonlinear data; performs segmented optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructs a visual distribution linear model based on the visual distribution linear data; replaces and fine-tunes the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model; generates a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0055] In an exemplary embodiment, as shown in FIG2 , a method for generating a hardware system model is provided, and the method is applied to the server 104 in FIG1 as an example for description, including the following steps S202 to S208 . Among them:

[0056] Step S202, data sampling is performed based on the distribution of the nonlinear activation function in the visual deformer to obtain visual distributed nonlinear data.

[0057] Among them, the visual deformer can adopt ViT (vision transformer), and the nonlinear activation function can be a Gelu activation function. Gelu refers to a Gaussian error linear unit, that is, a Gaussian error linear unit. The Gelu activation function is a nonlinear activation function commonly used in deep learning.

[0058] In some embodiments, by sampling the distribution of nonlinear activation functions in the visual deformer, visual distribution nonlinear data (piece wise linear) for establishing piecewise linear function PWL optimization is obtained, where PWL refers to piece wise linear.

[0059] Step S204, performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data.

[0060] Among them, segmented optimization refers to dividing the nonlinear data into multiple line segments, and transforming the nonlinear data into linear data with less loss and close to the original nonlinear data by optimization. Among them, visual distribution linear data refers to the relevant parameters of PWL.

[0061] In some embodiments, visual distribution nonlinear data is segmentedly optimized to obtain visual distribution linear data, including: obtaining hyperparameter data; segmented optimization setting of visual distribution nonlinear data according to the hyperparameter data to obtain visual distribution linear optimization data; evolutionarily updating the visual distribution linear optimization data based on an evolutionary model to obtain visual distribution linear data that meets preset conditions.

[0062] Among them, the hyper-parameter data can be hyper-parameter data such as the number of segments, and the preset conditions can be the loss size, the number of iterations, the iteration time, etc.

[0063] In some embodiments, hyper-parameter data may be acquired based on actual application requirements, or may be acquired based on data characteristics of visually distributed nonlinear data, but is not limited thereto.

[0064] In some embodiments, the visual distribution nonlinear data is segmented according to the hyperparameter data, and the visual distribution linear optimization data is set to obtain the visual distribution linear optimization data in the initial state; then, the visual distribution nonlinear data is further initialized and evolved by using the evolutionary model, and continuously updated candidate linear optimization data is obtained until the candidate linear optimization data meets the preset conditions, and the updating is stopped, and the candidate linear optimization data at this time is verified by MAE (mean absolute error). After the verification is passed, it is used as the visual distribution linear data, and the visual distribution linear data is exported for use in constructing a visual distribution linear model.

[0065] In some embodiments, a visual distribution linear model is constructed based on visual distribution linear data, including: performing differentiable activation processing based on the visual distribution linear data to obtain replaceable linear model data; wherein the replaceable linear model data is used for performing derivation processing in an unrestricted direction; and constructing a visual distribution linear model based on the replaceable linear model data.

[0066] Among them, the differentiable activation processing refers to differentiating the data into the corresponding activation function operator, the activation function operator is used to implement the activation function, and the replaceable linear model data refers to the corresponding activation function operator that is differentiable forward and backward.

[0067] In some embodiments, forward-differentiable or backward-differentiable activation processing is performed based on the visual distribution linear data to obtain replaceable linear model data that can be interchanged with the corresponding activation function operator for use in subsequent replacement fine-tuning processing. While cleverly replacing the operators in the original model to reduce the waste of computer resources, it can also be used for the subsequent painless export of model parameters.

[0068] Step S206, replacing and fine-tuning the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model.

[0069] The pre-trained original visual model may be a pre-trained visual model including a nonlinear gelu activation function.

[0070] In some embodiments, the pre-trained original visual model is replaced and fine-tuned based on the visual distribution linear model to obtain a target visual model, including: replacing the nonlinear sub-model of the original visual model with the visual distribution linear model to obtain the visual model to be fine-tuned; performing quantitative perceptual fine-tuning based on the visual model to be fine-tuned to obtain the target visual model.

[0071] The nonlinear sub-model refers to the nonlinear gelu activation function.

[0072] In some embodiments, before performing quantization perception fine-tuning, the nonlinear sub-model of the original visual model is replaced with a visual distribution linear model to obtain a visual model to be fine-tuned, and quantization perception training is performed on the visual model to be fine-tuned to obtain a quantized visual model. The quantized visual model is added to the visual model to be fine-tuned by a package addition method, so that the visual model to be fine-tuned and the quantized visual model are fine-tuned together to obtain a target visual model.

[0073] Among them, quantization-aware training refers to QAT (quantization-aware quantization).

[0074] Step S208, generating a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0075] Among them, generating hardware system processing refers to exporting the quantization parameters and the model in the model, and then generating the hardware system based on the exported data, so as to obtain an accurate and efficient hardware system model. The hardware system is an NPU system used for the deployment and reasoning process of neural networks. NPU refers to the neural network processor (Neural network Processing Unit).

[0076] It should be noted that the target hardware system model is generated through training based on the distribution of nonlinear activation functions in the visual deformer. Therefore, the target hardware system model is a hardware system model adapted to the visual deformer. In the process of the visual deformer processing and understanding images, the target hardware system model can perform inference and judgment through the PWL activation operator deployed in the target hardware system model, thereby ensuring the visual recognition accuracy and visual recognition performance of the visual deformer. Among them, the PWL activation operator refers to the PWL parameters derived from the trained target visual model, and the PWL activation operator is constructed based on the PWL parameters.

[0077] In some embodiments, a hardware system is generated based on a target visual model to obtain a target hardware system model, including: replacing a target linear sub-model format of the target visual model with an original nonlinear model format of the original visual model to obtain a target visual nonlinear model; and generating a hardware system based on quantization parameters of the target visual nonlinear model to obtain a target hardware system model.

[0078] The target linear sub-model format refers to the model format in the model file of the target linear sub-model.

[0079] In some embodiments, by replacing the target linear sub-model format of the target visual model with the original nonlinear model format of the original visual model, the model parameters of the target visual model obtained after training and fine-tuning are effectively retained, and only other structures and parameters except the activation function are saved, thereby obtaining a target visual nonlinear model that is easier to generate and deploy with hardware. Specifically, the differentiable gelu activation function of the target visual model can be replaced with a non-differentiable ordinary gelu activation function to facilitate the export of the model and model parameters, thereby being more conducive to the deployment and generation of the hardware system model. For example, when deploying, the fine-tuned middleware of the target visual model is imported into the hardware, and the hardware converts the corresponding activation function into an activation function that has been developed for the hardware, thereby completing an efficient deployment process.

[0080] In the above-mentioned hardware system model generation method, data sampling is performed based on the distribution of the nonlinear activation function in the visual deformer to obtain visual distribution nonlinear data; the visual distribution nonlinear data is segmentedly optimized to obtain visual distribution linear data, and a visual distribution linear model is constructed based on the visual distribution linear data; the pre-trained original visual model is replaced and fine-tuned based on the visual distribution linear model to obtain a target visual model; a hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer. Therefore, by segmenting and optimizing the visual distribution nonlinear data, linear data that is easy to calculate, namely, visual distribution linear data, is obtained, and then the hardware implementation process of the activation function is further optimized by replacement and fine-tuning, reducing the function calculation complexity and the cost and efficiency of hardware implementation. At the same time, the hardware system is generated based on the target visual model generated after fine-tuning, and the obtained target hardware system model can also guarantee the reasoning accuracy of the target hardware system model to a great extent.

[0081] In an exemplary embodiment, as shown in FIG3 , the hardware system model method further includes a performance evaluation step. Specifically, the performance evaluation step includes steps S302 to S304. Among them:

[0082] Step S302: Perform performance testing based on the target hardware system model to obtain performance testing data.

[0083] Among them, performance testing can be carried out by using specific reasoning scenarios or visual test sets to test the target hardware system model.

[0084] In some embodiments, the visual test set is subjected to inference testing through the target hardware system model to obtain test results, the test results are verified based on the visual verification set to obtain verification results, and the performance test data of the target hardware system model is determined according to the verification results. For example, if the accuracy rate is higher than a preset value, it means that the performance of the model reaches a certain rating, and the corresponding rating is used as its performance test data, or the prediction time, etc. can also be used as performance test data.

[0085] Step S304 , performing performance evaluation on the target visual linear model of the target hardware system model according to the performance test data to obtain performance evaluation data.

[0086] The performance evaluation data is performance evaluation data for a target visual linear model.

[0087] In some embodiments, the target visual linear model of the target hardware system model can be performance evaluated based on the performance test data to evaluate the prediction accuracy of the target visual linear model, i.e., the prediction precision. The average prediction time in the performance test data can reflect the performance of the target visual linear model, thereby obtaining accurate performance evaluation data for further optimization of the subsequent model training.

[0088] In order to more clearly understand the hardware system model generation process of the present application, it is described here in conjunction with FIG. 4, as follows:

[0089] In some embodiments, Figure 4 shows an embodiment of a specific hardware system model generation process. After sampling the distribution of the nonlinear activation function Gelu in the visual deformer, the obtained visual distribution nonlinear data is used to establish a segmented optimization process of the PWL model corresponding to the nonlinear activation function Gelu, and then the MAE error is verified until the visual distribution linear data that meets the preset conditions is obtained, and it is exported and plotted into a data distribution diagram as shown in Figure 4 to observe the model data more intuitively. Finally, the visual distribution linear data is exported. On the one hand, it can be used to develop a differentiable pwl Gelu, that is, a visual distribution linear model, and on the other hand, it can also be used to build an equivalent PWL activation operator in the hardware system to provide parameter support.

[0090] In some embodiments, Figure 4 also shows how to replace the Gelu activation function of the pre-trained original visual model with the differentiable pwl Gelu, i.e., the visual distributed linear model, to obtain the visual model to be fine-tuned, and then perform quantized perception processing on the visual model to be fine-tuned, and wrap the obtained quantized perception model with the original visual model to be fine-tuned and add them together for fine-tuning training. Due to the PWL activation function design and model quantization QAT, PWL is completed and QAT is introduced in a timely manner, and two steps of accuracy recovery are completed simultaneously in one fine-tuning, thereby ensuring the accuracy of the target visual model obtained, and the overall training time / training cost is low.

[0091] In some embodiments, FIG. 4 also shows that the pwl gelu activation function (i.e., the target linear sub-model format) in the target visual model is replaced with the gelu activation function. After two operator replacements, the model file still retains the original model format, and does not involve fine-tuning training and development of the corresponding PWL activation operator in the middle, but only saves other structures and parameters except the activation function. In this way, when deploying, the middleware after model fine-tuning is imported into the hardware, and the hardware converts the corresponding activation function into the activation that has been developed by the hardware, completing an efficient deployment process.

[0092] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0093] Based on the same inventive concept, the embodiment of the present application also provides a hardware system model generation device for implementing the hardware system model generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more hardware system model generation device embodiments provided below can refer to the limitations on the hardware system model generation method above, and will not be repeated here.

[0094] In an exemplary embodiment, as shown in FIG5 , a hardware system model generation device 500 is provided, comprising: a data sampling module 501, a segment optimization module 502, a replacement fine-tuning module 503 and a hardware system generation module 504, wherein:

[0095] A data sampling module 501 is used to perform data sampling based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data;

[0096] The segmented optimization module 502 is used to segmentally optimize the visual distribution nonlinear data to obtain visual distribution linear data, and to construct a visual distribution linear model according to the visual distribution linear data;

[0097] A replacement fine-tuning module 503 is used to replace and fine-tune the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model;

[0098] The hardware system generation module 504 is used to generate a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

[0099] In some embodiments, the segmented optimization module 502 is also used to obtain hyper-parameter data; perform segmented optimization settings on the visual distribution nonlinear data according to the hyper-parameter data to obtain visual distribution linear optimization data; and perform evolutionary updates on the visual distribution linear optimization data based on the evolutionary model to obtain visual distribution linear data that meets preset conditions.

[0100] In some embodiments, the segmented optimization module 502 is also used to perform derivable activation processing based on the visual distribution linear data to obtain replaceable linear model data; wherein the replaceable linear model data is used to perform derivation processing in an unrestricted direction; and a visual distribution linear model is constructed based on the replaceable linear model data.

[0101] In some embodiments, the replacement fine-tuning module 503 is further used to replace the nonlinear sub-model of the original visual model with a visual distribution linear model to obtain the visual model to be fine-tuned; and perform quantitative perceptual fine-tuning on the visual model to be fine-tuned to obtain a target visual model.

[0102] In some embodiments, the hardware system generation module 504 is also used to replace the target linear sub-model format of the target visual model with the original nonlinear model format of the original visual model to obtain the target visual nonlinear model; generate a hardware system based on the quantization parameters of the target visual nonlinear model to obtain a target hardware system model.

[0103] In some embodiments, the hardware system model generation device also includes: a performance testing module, which is used to perform performance testing based on the target hardware system model to obtain performance testing data; and to perform performance evaluation on the target visual linear model of the target hardware system model based on the performance test data to obtain performance evaluation data.

[0104] In the above hardware system model generation device, data sampling is performed based on the distribution of the nonlinear activation function in the visual deformer to obtain visual distribution nonlinear data; the visual distribution nonlinear data is segmented optimized to obtain visual distribution linear data, and a visual distribution linear model is constructed based on the visual distribution linear data; the pre-trained original visual model is replaced and fine-tuned based on the visual distribution linear model to obtain a target visual model; a hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer. Therefore, by segmenting and optimizing the visual distribution nonlinear data, linear data that is easy to calculate, namely, visual distribution linear data, is obtained, and then the hardware implementation process of the activation function is further optimized by replacement and fine-tuning, reducing the function calculation complexity and the cost and efficiency of hardware implementation. At the same time, the hardware system is generated based on the target visual model generated after fine-tuning, and the obtained target hardware system model can also guarantee the reasoning accuracy of the target hardware system model to a great extent.

[0105] Each module in the above hardware system model generation device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0106] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pre-trained original visual models. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating a hardware system model is implemented.

[0107] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned hardware system model generation method when executing the computer program.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned hardware system model generation method are implemented.

[0110] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned hardware system model generation method when executed by a processor.

[0111] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0112] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0113] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0114] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a hardware system model, characterized in that: The method comprises: Data sampling is performed based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data; Performing piecewise optimization on the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data; Based on the visual distribution linear model, the pre-trained original visual model is replaced and fine-tuned to obtain a target visual model; A hardware system is generated based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

2. The method according to claim 1, characterized in that The step of performing segmented optimization on the visual distribution nonlinear data to obtain visual distribution linear data includes: Get hyperparameter data; Performing segmented optimization setting on the visual distribution nonlinear data according to the hyperparameter data to obtain visual distribution linear optimization data; The visual distribution linear optimization data is evolutionarily updated based on the evolution model to obtain the visual distribution linear data that meets preset conditions.

3. The method according to claim 1, characterized in that The step of constructing a visual distribution linear model according to the visual distribution linear data comprises: Performing derivable activation processing according to the visual distribution linear data to obtain replaceable linear model data; wherein the replaceable linear model data is used for performing derivation processing in an unrestricted direction; The visual distribution linear model is constructed based on the replaceable linear model data.

4. The method according to claim 1, characterized in that: The replacing and fine-tuning the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model includes: Replacing the nonlinear sub-model of the original visual model with the visual distribution linear model to obtain a visual model to be fine-tuned; Quantization perception fine-tuning is performed based on the visual model to be fine-tuned to obtain the target visual model.

5. The method according to claim 1, characterized in that The step of generating a hardware system based on the target visual model to obtain a target hardware system model includes: Replacing the target linear sub-model format of the target visual model with the original nonlinear model format of the original visual model to obtain a target visual nonlinear model; A hardware system is generated based on the quantized parameters of the target visual nonlinear model to obtain the target hardware system model.

6. The method according to claim 1, characterized in that The method further comprises: Performing performance testing based on the target hardware system model to obtain performance testing data; A performance evaluation is performed on the target visual linear model of the target hardware system model according to the performance test data to obtain performance evaluation data.

7. A hardware system model generation device, characterized in that: The device comprises: A data sampling module, used for performing data sampling based on the distribution of nonlinear activation functions in the visual deformer to obtain visual distribution nonlinear data; A segmented optimization module, used for segmentally optimizing the visual distribution nonlinear data to obtain visual distribution linear data, and constructing a visual distribution linear model according to the visual distribution linear data; A replacement fine-tuning module, used for performing replacement fine-tuning on the pre-trained original visual model based on the visual distribution linear model to obtain a target visual model; The hardware system generation module is used to generate a hardware system based on the target visual model to obtain a target hardware system model; wherein the target hardware system model is used to control the visual deformer.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images