NLP model inference acceleration method and device based on elastic loading, equipment and medium

Abstract

The application relates to the technical field of natural language processing, and discloses an NLP model inference acceleration method based on elastic loading, which comprises the following steps: performing horizontal cutting on the hierarchical direction of a natural language model according to a horizontal cutting angle; extracting an attention matrix and a full connection layer in the natural language model, and performing vertical cutting on the attention matrix and the full connection layer according to a vertical cutting angle; calculating the hierarchical importance of a cutting level according to a target scene, and calculating the importance of a cutting slice according to the target scene; generating a submodel of the natural language model according to the hierarchical importance and the importance of the cutting slice; and interleaving and superimposing model loading tasks and inference calculation tasks in the submodel to perform the tasks. In addition, the application also relates to the blockchain technology, and medical scene data can be stored in the nodes of the blockchain. The application further provides an NLP model inference acceleration device, equipment and storage medium based on elastic loading. The application can improve the model inference efficiency and has great application prospects in the medical field.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, device, and medium for accelerating NLP model inference based on elastic loading. Background Technology

[0002] With the development of natural language technology, natural language models have achieved remarkable results in natural language processing. However, natural language models are too large, have long loading times, consume too much memory, and seriously affect deployability. In order to reduce model loading time and improve model inference speed, it is necessary to divide the completed model into multiple sub-models to accelerate model inference.

[0003] Existing model inference techniques load the entire model before performing the inference task. For example, when using rapid transcription prescription software to enter a patient's electronic medical record, to protect patient privacy, the software loads a natural language model on the terminal. However, execution only occurs after the entire natural language model is loaded, resulting in very low efficiency when entering the patient's electronic medical record and delaying the patient's diagnosis time. In practical applications, loading the entire model may lead to very slow response times, thus reducing efficiency during model inference. Summary of the Invention

[0004] This invention provides a method, apparatus, and computer-readable storage medium for accelerating NLP model inference based on elastic loading, with the main purpose of solving the problem of low efficiency in model inference.

[0005] To achieve the above objectives, this invention provides a method for accelerating NLP model inference based on elastic loading, comprising:

[0006] Obtain a preset natural language model, and segment the natural language model hierarchically by a preset horizontal segmentation angle to obtain the segmentation hierarchy;

[0007] The attention matrix and fully connected layer in the natural language model are extracted, and the attention matrix and fully connected layer are segmented by a preset vertical segmentation angle to obtain segmented pieces;

[0008] The hierarchy importance of the segmentation is calculated based on the preset target scenario, and the fragment importance of the segmentation is calculated based on the target scenario.

[0009] The natural language model is sub-modeled based on the hierarchical importance and the segmentation importance.

[0010] The model loading task and inference calculation task in the sub-model are executed in an interleaved manner to obtain the execution efficiency, and the inference acceleration of the natural language model is determined based on the execution efficiency.

[0011] Optionally, the step of segmenting the natural language model hierarchically by a preset horizontal segmentation angle to obtain the segmentation hierarchy includes:

[0012] Obtain the number of network layers of the natural language model;

[0013] The natural language model is segmented according to the horizontal segmentation angle and the number of network layers to obtain the segmentation hierarchy.

[0014] Optionally, the step of segmenting the attention matrix and the fully connected layer by a preset vertical segmentation angle to obtain segmented pieces includes:

[0015] The attention matrix is ​​segmented by the preset vertical segmentation angle to obtain a segmented attention matrix;

[0016] The fully connected layer is segmented according to the vertical segmentation angle to obtain a segmented fully connected layer.

[0017] The segmentation attention matrix and the segmentation fully connected layer are used to generate segmentation pieces.

[0018] Optionally, calculating the hierarchical importance of the segmentation levels based on a preset target scenario includes:

[0019] Determine the complete verification accuracy of the segmentation level based on the target scenario;

[0020] Determine the subset verification accuracy of the segmentation level based on the target scenario;

[0021] The hierarchical importance of the segmentation level is calculated based on the full verification accuracy and the subset verification accuracy using the following importance calculation formula:

[0022]

[0023] in, Indicates the importance of the hierarchy. This indicates the complete verification accuracy. This indicates the verification accuracy of the subset.

[0024] Optionally, determining the subset verification accuracy of the segmentation level based on the target scenario includes:

[0025] Delete any one of the cut-levels to obtain the molecular-level cut-level;

[0026] The molecular slicing hierarchy is divided into a training set and a test set;

[0027] The training set and the test set are validated using a preset cross-folding method to obtain the validation accuracy;

[0028] Calculate the average accuracy of the verification accuracy, and use the average accuracy as the subset verification accuracy of the segmentation level.

[0029] Optionally, generating sub-models of the natural language model based on the hierarchical importance and the fragment importance includes:

[0030] The importance of each level is sorted in descending order to obtain the first level of importance.

[0031] The importance of the fragments is sorted in descending order to obtain the second fragment importance.

[0032] Select the sub-level corresponding to the importance of the first level according to the preset first quantity threshold;

[0033] Select the sub-fragment corresponding to the importance of the second fragment according to the preset second quantity threshold;

[0034] The natural language model is generated based on the sub-level and the sub-segment.

[0035] Optionally, the step of interleaving the model loading task and the inference calculation task in the sub-model to obtain execution efficiency includes:

[0036] Obtain the first and second layers of the sub-model;

[0037] The loading task time is obtained by executing the model loading task in the first layer, and the calculation task time is obtained by executing the inference calculation task in the second layer.

[0038] The execution efficiency is calculated based on the loading task time and the computation task time.

[0039] To address the aforementioned problems, the present invention also provides an NLP model inference acceleration device based on elastic loading, the device comprising:

[0040] The horizontal segmentation module is used to obtain a preset natural language model and segment the natural language model hierarchically by a preset horizontal segmentation angle to obtain the segmentation hierarchy;

[0041] The vertical segmentation module is used to extract the attention matrix and fully connected layer from the natural language model, and to segment the attention matrix and fully connected layer by a preset vertical segmentation angle to obtain segmented pieces;

[0042] The fragment importance calculation module is used to calculate the hierarchical importance of the segmentation level according to the preset target scenario, and to calculate the fragment importance of the segmentation fragments according to the target scenario.

[0043] The sub-model generation module is used to generate sub-models of the natural language model based on the hierarchical importance and the piecewise importance.

[0044] The inference acceleration module is used to interleave the model loading task and the inference calculation task in the sub-model to obtain the execution efficiency, and to determine the inference acceleration of the natural language model based on the execution efficiency.

[0045] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0046] At least one processor; and,

[0047] A memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to execute the above-described method for accelerating NLP model inference based on elastic loading.

[0049] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned method for accelerating NLP model inference based on elastic loading.

[0050] This invention, through horizontal and vertical segmentation of the natural language model, divides it into multiple modules and calculates the importance of these modules across different datasets. For specific scenarios (medical datasets), high-importance modules are selected from the segmented hierarchical modules and chunks to form sub-models. By interleaving the model loading and inference tasks within these sub-models, latency is reduced during patient electronic medical record entry, and memory consumption is also decreased, significantly improving model inference speed and efficiency during electronic medical record processing. Therefore, the proposed method, device, electronic device, and computer-readable storage medium for accelerating NLP model inference based on elastic loading can solve the problem of low efficiency in model inference. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating an embodiment of the NLP model inference acceleration method based on elastic loading provided by the present invention.

[0052] Figure 2 This is a schematic diagram of the segmentation process provided in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the flowchart for calculating hierarchical importance provided in an embodiment of the present invention;

[0054] Figure 4 A functional block diagram of an NLP model inference acceleration device based on elastic loading provided in an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the NLP model inference acceleration method based on elastic loading, according to an embodiment of the present invention.

[0056] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0057] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0058] This application provides a method for accelerating NLP model inference based on elastic loading. The execution entity of this method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this application. In other words, the method can be executed by software or hardware installed on a terminal device or a server device, where the software may be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0059] Reference Figure 1 The diagram shown is a flowchart illustrating an NLP model inference acceleration method based on elastic loading according to an embodiment of the present invention. In this embodiment, the NLP model inference acceleration method based on elastic loading includes:

[0060] S1. Obtain a preset natural language model, and segment the natural language model hierarchically by a preset horizontal segmentation angle to obtain the segmentation hierarchy;

[0061] In one practical application scenario of this invention, a "note-taking" application software has many "intelligent text interaction functions". For example, in the "note-taking" software, doctors can use rapid transcription of prescriptions to enter electronic medical records (EHRs) or electronic health records. Natural language models can eliminate the need for doctors to sit in front of a computer at the nursing point and verbally recite prescriptions. At the same time, natural language models can correct patient problems before sending the transcribed text to the verification personnel.

[0062] In this embodiment of the invention, the Natural Language Processing (NLP) model is a standard Transformer model. The Transformer model is a model that uses an attention mechanism to improve the training speed of the model. It introduces a self-attention mechanism to learn all three types of relations. The Transformer proposes a multi-head attention mechanism to learn the corresponding three relations separately, using a full attention structure. For the position of words, the Transformer uses a positional encoding mechanism for data preprocessing, which increases the parallelism of the model and achieves better experimental results.

[0063] In detail, a pre-defined Natural Language Processing (NLP) model can be obtained by searching online. Let's assume that this network model has N layers and M attention matrices in each layer. Each layer contains M attention matrices and two fully connected layers (FFN1 and FFN2).

[0064] Furthermore, the model is segmented from two perspectives: the model is divided into multiple modules using "horizontal segmentation" and "vertical segmentation" methods, and the importance of each module is analyzed under different datasets.

[0065] In this embodiment of the invention, the step of segmenting the natural language model hierarchically by a preset horizontal segmentation angle to obtain segmentation levels includes:

[0066] Obtain the number of network layers of the natural language model;

[0067] The natural language model is segmented according to the horizontal segmentation angle and the number of network layers to obtain the segmentation hierarchy.

[0068] In detail, the horizontal segmentation angle is the segmentation according to the horizontal direction. If the number of network layers of the natural language model is N, then it can be divided into N modules, i.e. N segmentation levels, based on the layer.

[0069] Furthermore, vertical segmentation is performed using the attention matrix in the natural language model as the unit. Therefore, it is necessary to analyze the attention matrix in the natural language model.

[0070] S2. Extract the attention matrix and fully connected layer from the natural language model, and segment the attention matrix and fully connected layer by a preset vertical segmentation angle to obtain segmented pieces;

[0071] In this embodiment of the invention, the attention matrix and the fully connected layer are modules present in the natural language model. The attention matrix increases attention to regions of interest and suppresses the role of uninterested regions in image segmentation as much as possible. The fully connected layer acts as a "classifier" in the model, reassembling the previous local features into a complete image through a weight matrix.

[0072] In detail, the attention matrix and fully connected layers in the natural language model can be obtained through the model structure in the natural language model, wherein the number of network layers in the natural language model is N, each layer has M attention matrices, and each layer contains M attention matrices and two fully connected layers.

[0073] In this embodiment of the invention, the reference Figure 2 As shown, the step of segmenting the attention matrix and the fully connected layer by a preset vertical segmentation angle to obtain segmented pieces includes:

[0074] S21. The attention matrix is ​​segmented by the preset vertical segmentation angle to obtain a segmented attention matrix;

[0075] S22. The fully connected layer is segmented according to the vertical segmentation angle to obtain a segmented fully connected layer;

[0076] S23. Generate segmentation pieces from the segmentation attention matrix and the segmentation fully connected layer.

[0077] In detail, the vertical slicing angle is based on the attention matrix. According to the number of attention matrices, it can be divided into M blocks to obtain M slicing attention matrices. The fully connected layer is also sliced ​​according to the vertical slicing angle, and the two fully connected layers (FFN1 and FFN2) of each layer are also sliced ​​into M parts, thus obtaining M slicing fully connected layers.

[0078] Specifically, M segmentation attention matrices and M segmentation fully connected layers are combined to form M segmentation pieces, and each segmentation piece contains 1 attention matrix + FFN1+ FFN2.

[0079] Furthermore, the importance of each layer and each piece varies under different datasets. Therefore, it is necessary to analyze the importance of each layer and each piece in order to determine the composition of the sub-modules.

[0080] S3. Calculate the hierarchical importance of the segmentation level according to the preset target scenario, and calculate the fragment importance of the segmentation fragment according to the target scenario;

[0081] In this embodiment of the invention, the target scenario refers to different medical datasets. These medical datasets include medical records and patient inquiries from many patients. Therefore, the calculation of the hierarchical importance and the fragment importance differs under different medical datasets. It is necessary to calculate the accuracy of the importance of each layer and each fragment under different datasets.

[0082] In this embodiment of the invention, the reference Figure 3 As shown, calculating the hierarchical importance of the segmentation levels based on the preset target scenario includes:

[0083] S31. Determine the complete verification accuracy of the segmentation level based on the target scenario;

[0084] S32. Determine the subset verification accuracy of the segmentation level based on the target scenario;

[0085] S33. Calculate the hierarchical importance of the segmentation level based on the complete verification accuracy and the subset verification accuracy using the following importance calculation formula:

[0086]

[0087] in, Indicates the importance of the hierarchy. This indicates the complete verification accuracy. This indicates the verification accuracy of the subset.

[0088] In detail, for a given dataset, the full validation accuracy of all slice levels under the target scenario in the dataset is calculated in the complete model, and the subset validation accuracy of the sub-model missing that module in the specific scenario (dataset) is determined by discarding a slice level module.

[0089] Specifically, by removing the validation accuracy of a sub-model of a certain block in a specific scenario (dataset) from the validation accuracy of the complete model in a specific scenario (dataset), the hierarchical importance of each slice level can be determined.

[0090] In this embodiment of the invention, determining the subset verification accuracy of the segmentation level based on the target scene includes:

[0091] Delete any one of the cut-levels to obtain the molecular-level cut-level;

[0092] The molecular slicing hierarchy is divided into a training set and a test set;

[0093] The training set and the test set are validated using a preset cross-folding method to obtain the validation accuracy;

[0094] Calculate the average accuracy of the verification accuracy, and use the average accuracy as the subset verification accuracy of the segmentation level.

[0095] In detail, a certain level is discarded in the segmentation hierarchy to obtain a segmented molecular hierarchy. The segmented molecular hierarchy is then divided into training and test sets to verify the subset accuracy. The cross-folding refers to using k-fold cross-folding to divide the training and test sets, thereby obtaining different accuracies based on different divisions. The average accuracy obtained under all divisions is calculated, and the average accuracy is used as the subset verification accuracy of the segmentation hierarchy.

[0096] For example, the accuracy is validated using 5-fold cross-validation. The first fold is used as the test set, and the other folds 2-5 are used as the training set to obtain one accuracy. In turn, the second fold is used as the test set, and the other folds 1, 3, 4, and 5 are used as the training set, and so on, to obtain a total of 5 accuracies. The average of the 5 accuracies is taken as the validation accuracy to ensure the accuracy of the validation accuracy.

[0097] Furthermore, the step of determining the complete verification accuracy of the segmentation level based on the target scenario is consistent with the step of cross-folding the training set and the test set in the step of determining the subset verification accuracy of the segmentation level based on the target scenario, thereby obtaining the complete verification accuracy of the segmentation level.

[0098] In this embodiment of the invention, the step of calculating the importance of the segmented pieces based on the target scene is the same as the step of calculating the importance of the segmented levels based on the preset target scene, and will not be described again here.

[0099] Furthermore, based on the target scenario (dataset), important levels and important segments are selected from all the segmentation levels and segments to form a "sub-model," which improves the inference speed.

[0100] S4. Generate sub-models of the natural language model based on the hierarchical importance and the segmentation importance;

[0101] In this embodiment of the invention, the sub-model refers to selecting important levels and important pieces from all the original segmentation levels and all the original segmentation pieces to form a sub-model.

[0102] In this embodiment of the invention, generating a sub-model of the natural language model based on the hierarchical importance and the fragment importance includes:

[0103] The importance of each level is sorted in descending order to obtain the first level of importance.

[0104] The importance of the fragments is sorted in descending order to obtain the second fragment importance.

[0105] Select the sub-level corresponding to the importance of the first level according to the preset first quantity threshold;

[0106] Select the sub-fragment corresponding to the importance of the second fragment according to the preset second quantity threshold;

[0107] The natural language model is generated based on the sub-level and the sub-segment.

[0108] In detail, by arranging the calculated importance of each level in descending order, the first level importance of all segmentation levels can be obtained; similarly, by arranging the calculated importance of each piece in descending order, the second piece importance of all segmentation pieces can be obtained.

[0109] Specifically, multiple sub-levels are selected according to the first quantity threshold and the segmentation level corresponding to the first level importance; similarly, multiple sub-parts are selected according to the second quantity threshold and the segmentation level corresponding to the second part importance.

[0110] For example, select n important layer modules from N layers, and then select m fragments from M fragments within these n layers. The range of values ​​for n and m is then... , Where N represents the number of layers in the neural network, and M represents the number of attention matrices in each layer. These modules together form the "sub-model" we want.

[0111] Furthermore, the inference speed can be improved by combining sub-models, which requires interleaving and superimposing the tasks in the sub-models to improve the inference speed.

[0112] S5. The model loading task and inference calculation task in the sub-model are executed in an interleaved manner to obtain the execution efficiency, and the inference acceleration of the natural language model is determined based on the execution efficiency.

[0113] In one practical application scenario of this invention, when doctors record patients' electronic medical records, the natural language model needs to be fully loaded and converted into converted text for recording. This may result in slow conversion efficiency. Therefore, the model loading task and inference calculation task in the sub-model of the natural language model are interleaved and superimposed to improve the conversion efficiency of electronic medical records. Thus, the acceleration of natural language inference has great application prospects in the medical field.

[0114] In this embodiment of the invention, performing model loading and inference calculation tasks within a sub-model results in faster inference speed compared to loading the complete model before performing the inference task.

[0115] In this embodiment of the invention, the step of interleaving the model loading task and the inference calculation task in the sub-model to obtain execution efficiency includes:

[0116] Obtain the first and second layers of the sub-model;

[0117] The loading task time is obtained by executing the model loading task in the first layer, and the calculation task time is obtained by executing the inference calculation task in the second layer.

[0118] The execution efficiency is calculated based on the loading task time and the computation task time.

[0119] In detail, the sub-models are executed in an interleaved manner, that is, the first layer of the sub-model is used as the loading execution layer of the (i+1)th layer module, and the second layer of the sub-model is used as the data processing execution layer of the i-th layer module. Thus, while the model loading task is executed in the (i+1)th layer, the inference calculation task of the model is executed in the i-th layer.

[0120] Specifically, by recording the loading time of the model loading task when it is executed at layer i+1, and recording the computation time of the model inference computation task when it is executed at layer i, and based on the total execution time of the model inference computation task at layer i while the model loading task is executed at layer i+1, the total number of tasks in the sub-model is calculated. The task execution efficiency of a sub-model can be obtained by comparing the total number of modules with the total task execution time.

[0121] Furthermore, the inference acceleration of the natural language model is determined based on the execution efficiency, that is, the execution efficiency is compared with the execution efficiency of executing the inference task after loading the complete model. By interleaving the execution of tasks in the sub-models, the inference speed can be greatly improved, and not only is the latency reduced, but the memory consumption rate is also reduced.

[0122] This invention, through horizontal and vertical segmentation of the natural language model, divides it into multiple modules and calculates the importance of these modules across different datasets. For specific scenarios (medical datasets), high-importance modules are selected from the segmented hierarchical modules and chunks to form sub-models. By interleaving the model loading and inference tasks within these sub-models, latency is reduced during patient electronic medical record entry, and memory consumption is also decreased, significantly improving model inference speed and efficiency during electronic medical record processing. Therefore, the proposed method, device, electronic device, and computer-readable storage medium for accelerating NLP model inference based on elastic loading can solve the problem of low efficiency in model inference.

[0123] like Figure 4 The diagram shown is a functional block diagram of an NLP model inference acceleration device based on elastic loading provided in an embodiment of the present invention.

[0124] The NLP model inference acceleration device 100 based on elastic loading described in this invention can be installed in an electronic device. Depending on the functions implemented, the NLP model inference acceleration device 100 based on elastic loading may include a horizontal segmentation module 101, a vertical segmentation module 102, a segment importance calculation module 103, a sub-model generation module 104, and an inference acceleration module 105. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0125] In this embodiment, the functions of each module / unit are as follows:

[0126] The horizontal segmentation module 101 is used to obtain a preset natural language model and segment the hierarchical direction of the natural language model by a preset horizontal segmentation angle to obtain the segmentation hierarchy;

[0127] The vertical segmentation module 102 is used to extract the attention matrix and fully connected layer in the natural language model, and segment the attention matrix and fully connected layer by a preset vertical segmentation angle to obtain segmented pieces;

[0128] The fragment importance calculation module 103 is used to calculate the hierarchical importance of the segmentation level according to the preset target scenario, and to calculate the fragment importance of the segmentation fragments according to the target scenario.

[0129] The sub-model generation module 104 is used to generate sub-models of the natural language model based on the hierarchical importance and the piecewise importance.

[0130] The inference acceleration module 105 is used to interleave the model loading task and the inference calculation task in the sub-model to obtain the execution efficiency, and determine the inference acceleration of the natural language model based on the execution efficiency.

[0131] In detail, the modules in the NLP model inference acceleration device 100 based on elastic loading described in this embodiment of the invention employ the same methods as described above. Figures 1 to 3 The technique used is the same as the elastic loading-based NLP model inference acceleration method described in the article, and it can produce the same technical effect, so it will not be repeated here.

[0132] like Figure 5 The diagram shown is a structural schematic of an electronic device that implements an NLP model inference acceleration method based on elastic loading, according to an embodiment of the present invention.

[0133] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as an NLP model inference acceleration program based on elastic loading.

[0134] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing NLP model inference acceleration programs based on elastic loading) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0135] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as code for an NLP model inference acceleration program based on elastic loading, but also to temporarily store data that has been output or will be output.

[0136] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0137] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0138] The figure only shows an electronic device with components. Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0139] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0140] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0141] The NLP model inference acceleration program based on elastic loading, stored in the memory 11 of the electronic device 1, is a combination of multiple instructions that, when run in the processor 10, can achieve the following:

[0142] Obtain a preset natural language model, and segment the natural language model hierarchically by a preset horizontal segmentation angle to obtain the segmentation hierarchy;

[0143] The attention matrix and fully connected layer in the natural language model are extracted, and the attention matrix and fully connected layer are segmented by a preset vertical segmentation angle to obtain segmented pieces;

[0144] The hierarchy importance of the segmentation is calculated based on the preset target scenario, and the fragment importance of the segmentation is calculated based on the target scenario.

[0145] The natural language model is sub-modeled based on the hierarchical importance and the segmentation importance.

[0146] The model loading task and inference calculation task in the sub-model are executed in an interleaved manner to obtain the execution efficiency, and the inference acceleration of the natural language model is determined based on the execution efficiency.

[0147] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0148] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0149] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0150] Obtain a preset natural language model, and segment the natural language model hierarchically by a preset horizontal segmentation angle to obtain the segmentation hierarchy;

[0151] The attention matrix and fully connected layer in the natural language model are extracted, and the attention matrix and fully connected layer are segmented by a preset vertical segmentation angle to obtain segmented pieces;

[0152] The hierarchy importance of the segmentation is calculated based on the preset target scenario, and the fragment importance of the segmentation is calculated based on the target scenario.

[0153] The natural language model is sub-modeled based on the hierarchical importance and the segmentation importance.

[0154] The model loading task and inference calculation task in the sub-model are executed in an interleaved manner to obtain the execution efficiency, and the inference acceleration of the natural language model is determined based on the execution efficiency.

[0155] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0156] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0158] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0159] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0160] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0161] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0162] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An elastic loading-based NLP model inference acceleration method, characterized in that, The method comprises: acquiring a preset natural language model, performing horizontal direction cutting on the natural language model by a preset horizontal cutting angle to obtain a cutting level; extracting an attention matrix and a full connection layer of each layer in the natural language model, and performing cutting on the attention matrix and the full connection layer by a preset vertical cutting angle to obtain a cutting slice; calculating a level importance of the cutting level according to a preset medical data set, and calculating a slice importance of the cutting slice according to the medical data set, wherein the medical data set comprises medical records and inquiry questions of patients; generating a sub-model of the natural language model according to the level importance and the slice importance, wherein the sub-model comprises n*m modules, n is the number of layers of the module, and m is the number of slices contained in each layer of the module; interleaving and superimposing model loading tasks and inference calculation tasks in the sub-model to obtain an execution efficiency, and determining inference acceleration of the natural language model according to the execution efficiency; the interleaving and superimposing of the model loading tasks and the inference calculation tasks in the sub-model to obtain the execution efficiency comprises: acquiring a first layer and a second layer in the sub-model; performing the model loading task through the first layer to obtain a loading task time, and performing the inference calculation task through the second layer to obtain a calculation task time; calculating the execution efficiency according to the loading task time and the calculation task time; taking the first layer in the sub-model as an i+1 layer, and taking the second layer in the sub-model as an i layer, then performing the model loading task in the i+1 layer while performing the inference calculation task in the i layer.

2. The elastic load-based NLP model inference acceleration method of claim 1, wherein, the horizontal direction cutting of the natural language model by the preset horizontal cutting angle to obtain the cutting level comprises: acquiring the number of network layers of the natural language model; cutting the level of the natural language model according to the horizontal cutting angle and the number of network layers to obtain the cutting level.

3. The elastic load-based NLP model inference acceleration method of claim 1, wherein, the cutting of the attention matrix and the full connection layer by the preset vertical cutting angle to obtain the cutting slice comprises: cutting the attention matrix by the preset vertical cutting angle to obtain a cutting attention matrix; cutting the full connection layer according to the vertical cutting angle to obtain a cutting full connection layer; generating the cutting slice from the cutting attention matrix and the cutting full connection layer.

4. The elastic load-based NLP model inference acceleration method of claim 1, wherein, the calculation of the level importance of the cutting level according to the preset medical data set comprises: determining a complete verification accuracy of the cutting level according to the medical data set; determining a subset verification accuracy of the cutting level according to the medical data set; calculating the level importance of the cutting level according to the complete verification accuracy and the subset verification accuracy by using the following importance calculation formula: wherein, denotes the hierarchical importance, denotes the full verification accuracy, denotes the subset verification accuracy.

5. The elastic load-based NLP model inference acceleration method of claim 4, wherein, the determination of the subset verification accuracy of the cutting level according to the medical data set comprises: deleting any optional level in the cutting level to obtain a cutting sub-level; dividing the cutting sub-level into a training set and a test set; verifying the training set and the test set by using a preset cross-validation to obtain a verification accuracy; An average precision of the verification precision is calculated, and the average precision is used as the subset verification precision of the split level.

6. The elastic load-based NLP model inference acceleration method according to any one of claims 1 to 5, wherein, The generating the sub-model of the natural language model according to the level importance and the split importance comprises: The level importance is sorted in descending order to obtain a first level importance; The split importance is sorted in descending order to obtain a second split importance; A sub-level corresponding to the first level importance is selected according to a preset first quantity threshold; A sub-split corresponding to the second split importance is selected according to a preset second quantity threshold; The sub-model of the natural language model is generated according to the sub-level and the sub-split.

7. An elastic load-based NLP model inference acceleration apparatus, characterized in that, The device comprises: A horizontal split module is configured to obtain a preset natural language model, split the level direction of the natural language model by a preset horizontal split angle, and obtain a split level. A vertical split module is configured to extract an attention matrix and a full connection layer of each layer in the natural language model, split the attention matrix and the full connection layer by a preset vertical split angle, and obtain a split split. A split importance calculation module is configured to calculate a level importance of the split level according to a preset medical data set, and calculate a split importance of the split split according to the medical data set, wherein the medical data set comprises medical records and inquiry questions of patients. A sub-model generation module is configured to generate a sub-model of the natural language model according to the level importance and the split importance, wherein the sub-model comprises n*m modules, n is the number of module layers, and m is the number of splits included in each layer of modules. An inference acceleration module is configured to interleave and superimpose the model loading task and the inference calculation task in the sub-model to obtain an execution efficiency, and determine the inference acceleration of the natural language model according to the execution efficiency.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the NLP model inference acceleration method based on elastic loading according to any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the NLP model inference acceleration method based on elastic loading according to any one of claims 1 to 6.

Images