Request processing method and device based on neural network model

Abstract

The present invention discloses a request processing method and device based on a neural network model, which relates to the field of computer technology and can solve the problem in the prior art that hardware resources cannot change rapidly and dynamically with changes in traffic and have poor flexibility. The method of the present invention mainly includes: determining whether at least one target neural network model is stored in the video memory of a network processor, the target neural network model including a neural network model corresponding to a queue of pending requests; if the target neural network model is not stored in the video memory, pre-fetching the target neural network model from the internal memory and loading it into the video memory to cover at least one neural network model pre-stored in the video memory; processing the requests in the queue of pending requests based on the target neural network model stored in the video memory. The present invention is mainly applicable to scenarios in which different requests are processed based on multiple neural network models.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a request processing method based on a neural network model. Background Art

[0002] With the rapid development of computer technology, neural network models have been gradually applied to various fields such as computer vision and natural language processing, and have achieved good results. And with the increase in data processing needs, the scale of neural network models that need to be deployed is getting larger and larger. For example, a data processing needs to support translation functions in more than 20 languages. Each language direction (such as from Chinese to English) requires at least one model to provide services. If so many languages ​​support mutual translation, then the system needs to support more than 400 models.

[0003] Each neural network model needs to be loaded into the network processor (Neural-network Processing Unit, NPU for short) when it is executed, but the video memory of a single network processor is very limited, so it is impossible to deploy so many neural network models on the same network processor. In order to support the service of large-scale models, the current method is to deploy multiple network processors to form a service cluster, and each network processor is only responsible for loading a part of the neural network model (for example, responsible for loading 10-20 models). However, hardware resources cannot change quickly and dynamically with changes in traffic. For example, when the traffic increases in a short period of time, it is impossible to quickly add a new network processor to share the traffic, and the flexibility is poor. Summary of the invention

[0004] In view of this, the request processing method and device based on the neural network model provided by the present invention aims to solve the problem in the prior art that hardware resources cannot change rapidly and dynamically with changes in traffic and have poor flexibility.

[0005] In a first aspect, the present invention provides a request processing method based on a neural network model, the method comprising:

[0006] Determining whether at least one target neural network model is stored in a video memory of a network processor, the target neural network model including a neural network model corresponding to a queue of pending requests;

[0007] If the target neural network model is not stored in the video memory, pre-fetching the target neural network model from the memory and loading it into the video memory to overwrite at least one neural network model pre-stored in the video memory;

[0008] The requests in the pending request queue are processed based on the target neural network model stored in the video memory.

[0009] Optionally, pre-fetching the target neural network model from the memory and loading it into the video memory to cover at least one neural network model pre-stored in the video memory includes:

[0010] Pre-fetching the target neural network model from the memory;

[0011] Determining whether the video memory contains a neural network model that is not executed;

[0012] If the video memory contains a neural network model that is not executed, then based on the overwriting strategy, the target neural network model overwrites at least one neural network model that is not executed.

[0013] Optionally, the method further includes:

[0014] If there is no neural network model in an unexecuted state in the video memory, determining whether the remaining space size of the video memory is greater than or equal to the size of the target neural network model;

[0015] If the size of the remaining space is greater than or equal to the size of the target neural network model, the target neural network model is directly loaded into the video memory.

[0016] Optionally, based on the coverage strategy, making the target neural network model cover at least one neural network model in an unexecuted state includes:

[0017] If the reserved space of the video memory is greater than a preset space threshold, a neural network model that is not executed is randomly selected, and the target neural network model is used to overwrite the selected neural network model;

[0018] If the reserved space of the video memory is less than a preset space threshold, the target neural network model is used to overwrite at least one of the neural network models in the unexecuted state based on the size of the neural network model in the unexecuted state and the size of the target neural network model.

[0019] Optionally, pre-fetching the target neural network model from the memory and loading it into the video memory to cover at least one neural network model pre-stored in the video memory includes:

[0020] The prefetch thread notifies the network processor to prefetch the target neural network model from the memory and load it into the video memory to overwrite at least one neural network model pre-stored in the video memory.

[0021] Optionally, processing the requests in the pending request queue based on the target neural network model stored in the video memory includes:

[0022] The prefetch thread adds the name of the target neural network model to the prefetch task queue;

[0023] The pre-fetch thread notifies the working thread to process the request in the request queue corresponding to the name of the target neural network model in the pre-fetch task queue.

[0024] Optionally, after the prefetch thread adds the name of the target neural network model to the prefetch task queue, the method further includes:

[0025] If the number of names contained in the pre-fetch task queue is greater than or equal to a preset number threshold, the pre-fetch thread waits for the number of names contained in the pre-fetch task queue to decrease to less than the preset number threshold, and then processes the target neural network model corresponding to the next pending request queue;

[0026] If the number of names contained in the pre-fetch task queue is less than a preset number threshold, the pre-fetch thread processes the target neural network model corresponding to the next pending request queue.

[0027] Optionally, the determining whether at least one target neural network model is stored in a video memory of a network processor includes:

[0028] If there are requests in the pending request queue, it is determined whether at least one target neural network model corresponding to the pending request queue is stored in the video memory.

[0029] In a second aspect, the present invention provides a request processing method based on a neural network model, the method comprising:

[0030] Determining whether at least one target language translation model is stored in a video memory of a graphics processor, the target language translation model comprising a neural network model for language translation corresponding to the pending request queue;

[0031] If the target language translation model is not stored in the video memory, pre-fetching the target language translation model from the internal memory and loading it into the video memory to overwrite at least one language translation model pre-stored in the video memory;

[0032] Processing the requests in the pending request queue based on the target language translation model stored in the video memory;

[0033] Output the translation results corresponding to the requests in the pending request queue.

[0034] In a third aspect, the present invention provides a request processing method based on a neural network model, the method comprising:

[0035] receiving a request sent by a requesting party;

[0036] Determining, based on the request, at least one target neural network model required to process the request;

[0037] Determining whether the target neural network model is stored in the first storage space;

[0038] If the target neural network model is not stored in the first storage space, pre-fetching the target neural network model from the second storage space and loading it into the first storage space to overwrite at least one neural network model pre-stored in the first storage space;

[0039] Processing the request based on the target neural network model stored in the first storage space;

[0040] The processing result is sent to the requesting party.

[0041] In a fourth aspect, the present invention provides a request processing device based on a neural network model, the device comprising:

[0042] A determination unit, configured to determine whether at least one target neural network model is stored in a video memory of a network processor, wherein the target neural network model includes a neural network model corresponding to a queue of pending requests;

[0043] A loading unit, configured to pre-fetch the target neural network model from the memory and load it into the video memory if the target neural network model is not stored in the video memory, so as to overwrite at least one neural network model pre-stored in the video memory;

[0044] A request processing unit is used to process the requests in the pending request queue based on the target neural network model stored in the video memory.

[0045] Optionally, the loading unit includes:

[0046] A pre-fetch module, used for pre-fetching the target neural network model from the memory;

[0047] A judgment module, used for judging whether the video memory contains a neural network model in an unexecuted state;

[0048] An overwriting module is used to make the target neural network model overwrite at least one neural network model in an unexecuted state based on an overwriting strategy if the video memory contains a neural network model in an unexecuted state.

[0049] Optionally, the judging unit is further configured to judge whether the remaining space size of the video memory is greater than or equal to the size of the target neural network model if there is no neural network model in an unexecuted state in the video memory;

[0050] The loading unit is also used to load the target neural network model directly into the video memory if the size of the remaining space is greater than or equal to the size of the target neural network model.

[0051] Optionally, the overwriting module is used to randomly select a neural network model in an unexecuted state and use the target neural network model to overwrite the selected neural network model if the reserved space of the video memory is larger than a preset space threshold; if the reserved space of the video memory is smaller than the preset space threshold, use the target neural network model to overwrite at least one neural network model in an unexecuted state based on the size of the neural network model in the unexecuted state and the size of the target neural network model.

[0052] Optionally, the loading unit is used to notify the network processor by a pre-fetch thread to pre-fetch the target neural network model from the memory and load it into the video memory to overwrite at least one neural network model pre-stored in the video memory.

[0053] Optionally, the request processing unit includes:

[0054] An adding module, configured to add the name of the target neural network model to a pre-fetch task queue by a pre-fetch thread;

[0055] A notification processing module is used for the pre-fetch thread to notify the working thread to process the request in the request queue corresponding to the name of the target neural network model in the pre-fetch task queue.

[0056] Optionally, the device further comprises:

[0057] A waiting processing unit, configured to process the target neural network model corresponding to the next pending request queue after the prefetch thread adds the name of the target neural network model to the prefetch task queue, if the number of names contained in the prefetch task queue is greater than or equal to a preset number threshold, and then wait for the number of names contained in the prefetch task queue to decrease to less than the preset number threshold;

[0058] The processing unit is used to process the target neural network model corresponding to the next pending request queue if the number of names contained in the pre-fetch task queue is less than a preset number threshold.

[0059] Optionally, the judgment unit is used to judge whether at least one target neural network model corresponding to the pending request queue is stored in the video memory if there is a request in the pending request queue.

[0060] In a fifth aspect, the present invention provides a request processing device based on a neural network model, the device comprising:

[0061] A determination unit, configured to determine whether at least one target language translation model is stored in a video memory of a graphics processor, wherein the target language translation model includes a neural network model for language translation corresponding to the pending request queue;

[0062] a loading unit, configured to pre-fetch the target language translation model from the internal memory and load it into the video memory if the target language translation model is not stored in the video memory, so as to overwrite at least one language translation model pre-stored in the video memory;

[0063] a request processing unit, configured to process requests in the pending request queue based on the target language translation model stored in the video memory;

[0064] The output unit is used to output the translation results corresponding to the requests in the pending request queue.

[0065] In a sixth aspect, the present invention provides a request processing device based on a neural network model, the device comprising:

[0066] A receiving unit, configured to receive a request sent by a requesting party;

[0067] a determination unit, configured to determine, according to the request, at least one target neural network model required to process the request;

[0068] A judging unit, used to judge whether the target neural network model is stored in the first storage space;

[0069] A loading unit, configured to pre-fetch the target neural network model from the second storage space and load it into the first storage space if the target neural network model is not stored in the first storage space, so as to overwrite at least one neural network model pre-stored in the first storage space;

[0070] a processing unit, configured to process the request based on the target neural network model stored in the first storage space;

[0071] The sending unit is used to send the processing result to the requesting party.

[0072] In a seventh aspect, the present invention provides a storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the request processing method based on the neural network model as described in the first aspect or the second aspect.

[0073] In an eighth aspect, the present invention provides an electronic device, the electronic device comprising a storage medium and a processor;

[0074] The processor is adapted to implement each instruction;

[0075] The storage medium is suitable for storing a plurality of instructions;

[0076] The instructions are suitable for being loaded by the processor and executing the request processing method based on the neural network model as described in the first aspect or the second aspect.

[0077] By means of the above technical scheme, the request processing method and device based on the neural network model provided by the present invention can pre-store at least one neural network model in a memory whose storage space is much larger than the video memory. When it is necessary to use at least one target neural network model corresponding to the pending request queue to process the request, if the target neural network model is not stored in the video memory of the network processor, the target neural network model can be pre-fetched from the memory, and by overwriting at least one neural network model pre-stored in the video memory, the target neural network model can be successfully loaded into the video memory to process the request, and the target neural network model will not be unable to be loaded due to insufficient video memory space, thereby indirectly expanding the video memory through the memory, and then realizing the function of loading large-scale neural network models using a single network processor, and can dynamically change rapidly with changes in traffic through software resources, thereby improving flexibility.

[0078] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0080] Figure 1 A flowchart of a request processing method based on a neural network model provided by an embodiment of the present invention is shown;

[0081] Figure 2 A flowchart of another request processing method based on a neural network model provided by an embodiment of the present invention is shown;

[0082] Figure 3 A flow chart of a method for processing requests based on a language translation model provided by an embodiment of the present invention is shown;

[0083] Figure 4 An example diagram of request processing based on a neural network model provided by an embodiment of the present invention is shown;

[0084] Figure 5 A block diagram showing a request processing device based on a neural network model provided by an embodiment of the present invention;

[0085] Figure 6 A block diagram showing another request processing device based on a neural network model provided by an embodiment of the present invention;

[0086] Figure 7 A block diagram showing a composition of another request processing device based on a neural network model provided by an embodiment of the present invention;

[0087] Figure 8 A block diagram showing the composition of another request processing device based on a neural network model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0089] In the case of needing to deploy a large-scale neural network model, a service cluster can be constructed through multiple network processors, so that multiple network processors can load a part of the neural network model respectively to achieve this. However, this hardware deployment method is limited by the inflexibility of resource mobilization and allocation. In order to solve this technical problem, an embodiment of the present invention provides a request processing method based on a neural network model. This method uses a relatively large memory to temporarily store the required neural network model. When the video memory is needed, the neural network model stored in the memory is loaded into the video memory, so that only a single network processor is needed to realize the indirect expansion function of the video memory. Figure 1 As shown, the specific implementation process of the method includes:

[0090] 101. Determine whether at least one target neural network model is stored in a video memory of a network processor.

[0091] The target neural network model includes the neural network model corresponding to the pending request queue. NPU includes but is not limited to a graphics processing unit (GPU), a TPU, a tensor processing unit (TPU), etc.

[0092] When a new request is received, it is possible to first determine which neural network model (one or more) is needed to process the request based on the content in the request (which may be a model identifier), and add the request to the request queue corresponding to the determined neural network model to wait for processing. Since the embodiment of the present invention temporarily stores some neural network models through memory, when the neural network model needs to be used, it still needs to be loaded into the video memory. Therefore, in order to improve the processing efficiency of the request, the required neural network model can be loaded into the video memory in advance so that it can be used directly when the neural network model is needed. A request queue can correspond to one neural network model or multiple neural network models. For example, some requests require multiple neural network models to be processed in sequence.

[0093] Specifically, all request queues can be polled first; determine whether there are requests in the currently polled pending request queue; if there are requests in the pending request queue, determine whether at least one target neural network model corresponding to the pending request queue is stored in the video memory, and determine whether the target neural network model needs to be preloaded based on the determination result; if there are no requests in the pending request queue, there is no need to process the pending request queue, and continue to poll the next request queue.

[0094] 102. If the target neural network model is not stored in the video memory, pre-fetch the target neural network model from the internal memory and load it into the video memory to overwrite at least one neural network model pre-stored in the video memory.

[0095] When the network processor is initially used, there is sufficient storage space in its video memory, and at least one neural network model to be used can be directly loaded into the video memory for use; the neural network model that needs to be used later can be stored in the memory first, and when the target neural network model in the memory is to be used, in order to successfully use the target neural network model, the target neural network model can be pre-fetched from the memory and loaded into the video memory to overwrite the at least one neural network model pre-stored in the video memory.

[0096] Among them, the specific implementation method of pre-fetching the target neural network model from the memory and loading it into the video memory to cover at least one neural network model pre-stored in the video memory can be: pre-fetching the target neural network model from the memory; judging whether the video memory contains a neural network model in an unexecuted state; if it contains a neural network model in an unexecuted state, based on the coverage strategy, making the target neural network model cover at least one neural network model in an unexecuted state; if there is no neural network model in an unexecuted state in the video memory (that is, all neural network models in the video memory are in an executed state), it is possible to wait for the appearance of an executed neural network model in the video memory before covering it. In order to make full use of the storage space of the video memory, if there is no neural network model in an unexecuted state in the video memory, it is also possible to first judge whether the remaining space size of the video memory is greater than or equal to the size of the target neural network model. If the remaining space size is greater than or equal to the size of the target neural network model, the target neural network model is directly loaded into the video memory; if the remaining space size is less than the size of the target neural network model, it is possible to wait for the appearance of an executed neural network model in the video memory, and then load the target neural network model by covering other neural network models.

[0097] In order to make full use of the storage space of the video memory, when it is determined that the target neural network model is not stored in the video memory, it is also possible to first determine whether the remaining space size of the video memory is greater than or equal to the size of the target neural network model; if the remaining space size of the video memory is greater than or equal to the size of the target neural network model, then the target neural network model is pre-fetched from the memory and then directly loaded into the video memory; if the remaining space size of the video memory is less than the size of the target neural network model, then the target neural network model is pre-fetched from the memory and loaded into the video memory to overwrite at least one neural network model pre-stored in the video memory.

[0098] Based on the overwriting strategy, the specific implementation method of making the target neural network model overwrite at least one neural network model in an unexecuted state can be: if the reserved space of the video memory is greater than the preset space threshold, then a neural network model in an unexecuted state is randomly selected, and the target neural network model is used to overwrite the selected neural network model, or a neural network model whose execution order is after the target neural network model is selected from the neural network models in an unexecuted state is overwritten by the target neural network model; if the reserved space of the video memory is less than the preset space threshold, then according to the size of the neural network model in an unexecuted state and the size of the target neural network model, the target neural network model is used to overwrite at least one neural network model in an unexecuted state. Among them, the reserved space is not the real-time remaining space of the video memory, but the storage space reserved in advance when loading the neural network model.

[0099] For example, if the video memory can store 5 neural network models, in the specific implementation, it is set to store only 3 neural network models at most, and reserve space for 2 neural network models. In this case, no matter how large the next target neural network model is, a neural network model is randomly selected to be overwritten, and the target neural network model can be successfully loaded.

[0100] When the reserved space is smaller than the preset space threshold, random coverage may not be achieved, and it is necessary to select which neural network model to cover based on the actual situation. Specifically, the size of each unexecuted neural network model and the size of the target neural network model can be determined first; if there is an unexecuted neural network model whose size is greater than or equal to the target neural network model, the target neural network model can be used to directly cover the neural network model; if the sizes of all unexecuted neural network models are smaller than the size of the target neural network model, the target neural network model can be used to cover the sizes of two of the neural network models.

[0101] 103. Process the requests in the pending request queue based on the target neural network model stored in the video memory.

[0102] When the target neural network model is pre-loaded into the video memory, when the target neural network model is needed, the target neural network model can be called to process the corresponding requests one by one.

[0103] The neural network model-based request processing method provided in an embodiment of the present invention can pre-store at least one neural network model in a memory whose storage space is much larger than the video memory. When it is necessary to use at least one target neural network model corresponding to the pending request queue to process the request, if the target neural network model is not stored in the video memory of the network processor, the target neural network model can be pre-fetched from the memory, and by overwriting at least one neural network model pre-stored in the video memory, the target neural network model can be successfully loaded into the video memory to process the request without causing the target neural network model to be unable to be loaded due to insufficient video memory space. Indirect expansion of the video memory is achieved through the memory, thereby realizing the function of loading large-scale neural network models using a single network processor, and can dynamically change rapidly with changes in traffic through software resources, thereby improving flexibility.

[0104] Optional, Figure 1 The method embodiment shown can be completed by a pre-fetch thread and a working thread, such as Figure 2 As shown, the specific implementation process includes:

[0105] 201. The pre-fetch thread determines whether at least one target neural network model is stored in the video memory of the network processor; if it is stored in the video memory, execute step 202; if it is not stored in the video memory, execute steps 203-205.

[0106] Specifically, the prefetch thread polls all request queues; the prefetch thread determines whether the currently polled pending request queue contains any request; if it contains any request, the prefetch thread determines whether at least one target neural network model is stored in the video memory of the network processor; if there is no request, the prefetch thread continues to poll the next request queue.

[0107] The specific implementation method of the pre-fetch thread determining whether at least one target neural network model is stored in the video memory of the network processor is the same as the above step 101 and will not be repeated here.

[0108] 202. The pre-fetch thread notifies the working thread to process the requests in the pending request queue based on the target neural network model.

[0109] The embodiment of the present invention can use one working thread to process the request queues corresponding to different neural network models through time division multiplexing; or it can start multiple working threads, and different working threads respectively process the request queues corresponding to different neural network models.

[0110] 203. The prefetch thread notifies the network processor to prefetch the target neural network model from the memory and load it into the video memory to overwrite at least one neural network model pre-stored in the video memory.

[0111] The specific implementation method of the network processor pre-fetching the target neural network model from the memory and loading it into the video memory to cover at least one neural network model pre-stored in the video memory is the same as the above step 102 and will not be repeated here.

[0112] 204. The pre-fetch thread adds the name of the target neural network model to the pre-fetch task queue.

[0113] Since the target neural network model is pre-loaded into the video memory and it may take some time before it is used, the name of the target neural network model can be added to the pre-fetch task queue after loading the target neural network model into the video memory, so that subsequent working threads can determine which neural network model's corresponding request queue needs to be processed based on the pre-fetch task queue.

[0114] In actual applications, while the working thread processes the request queue, the pre-fetch thread will perform the neural network model pre-loading process. If there are many requests in the request queue, the working thread will take a relatively long time to process the request queue, and the number of neural network models pre-loaded by the pre-fetch thread will also increase, resulting in the problem that the pre-loading speed is much greater than the request processing speed, which will cause the waste of pre-fetch thread resources. In order to avoid the waste of pre-fetch thread resources, after the pre-fetch thread adds the name of the target neural network model to the pre-fetch task queue, it can be determined whether the number of names contained in the pre-fetch task queue is greater than or equal to the preset number threshold; if the number of names contained in the pre-fetch task queue is greater than or equal to the preset number threshold, the pre-fetch thread waits for the number of names contained in the pre-fetch task queue to decrease to less than the preset number threshold, and then processes the target neural network model corresponding to the next request queue to be processed; if the number of names contained in the pre-fetch task queue is less than the preset number threshold, the pre-fetch thread processes the target neural network model corresponding to the next request queue to be processed. Among them, the preset number threshold can be 2, or it can be other values, which depends on actual experience.

[0115] 205. The pre-fetch thread notifies the working thread to process the request in the request queue corresponding to the name of the target neural network model in the pre-fetch task queue.

[0116] The working thread determines whether the pre-fetch task queue contains the name of the neural network model; if it contains the name, it can obtain a batch of requests from the request queue corresponding to the name, and submit the obtained requests to the network processor for processing based on the target neural network model. After the processing is completed, the working thread notifies the pre-fetch thread and returns the processing result.

[0117] The neural network model-based request processing method provided in an embodiment of the present invention can pre-fetch at least one target neural network model from the memory and load it into the video memory by a pre-fetch thread to overwrite at least one neural network model pre-stored in the video memory, and notify the working thread to process the request queue corresponding to the target neural network model without causing the target neural network model to be unable to be loaded due to insufficient video memory space, thereby achieving indirect expansion of the video memory through the memory, and further realizing the function of loading large-scale neural network models using a single network processor, and can dynamically change rapidly with changes in traffic through software resources, thereby improving flexibility.

[0118] Furthermore, the above method embodiment can be applied to a variety of scenarios, such as language translation scenarios, image processing scenarios, etc. The following takes the language translation scenario as an example to illustrate the above method. Figure 3 As shown, the method includes:

[0119] 301. Determine whether at least one target language translation model is stored in a video memory of a graphics processor.

[0120] The target language translation model includes a neural network model corresponding to the pending request queue and used for language translation.

[0121] 302. If the target language translation model is not stored in the video memory, pre-fetch the target language translation model from the internal memory and load it into the video memory to overwrite at least one language translation model pre-stored in the video memory.

[0122] 303. Process the requests in the current request queue based on the target language translation model stored in the video memory.

[0123] The specific implementation of steps 301-303 can be referred to Figure 1 Or the embodiment shown in 2 is not repeated here.

[0124] 304. Output the translation results corresponding to the requests in the pending request queue.

[0125] The embodiment of the present invention does not limit the output method of the translation result. The request can be output on the left side of the page, and the translation result can be output on the right side of the page, and it is marked which request the translation result is the translation result of.

[0126] The neural network model-based request processing method provided in an embodiment of the present invention can pre-store at least one language translation model in a memory whose storage space is much larger than the video memory. When it is necessary to use at least one target language translation model corresponding to the pending request queue to process the request, if the target language translation model is not stored in the video memory of the network processor, the target neural network model can be pre-fetched from the memory, and by overwriting at least one language translation model pre-stored in the video memory, the target language translation model can be successfully loaded into the video memory to process the request and output the translation result without causing the target language translation model to be unable to be loaded due to insufficient video memory space. In this way, indirect expansion of the video memory is achieved through the memory, and the function of loading large-scale language translation models using a single network processor is realized. In addition, software resources can be dynamically changed rapidly with changes in traffic, thereby improving flexibility.

[0127] Further, according to the above method embodiment, another embodiment of the present invention also provides a request processing method based on a neural network model, the method is applied to a receiving party, and the method includes:

[0128] S1. Receive a request sent by a requesting party;

[0129] S2. Determine at least one target neural network model required to process the request according to the request; when the request includes a model identifier, the target neural network model can be determined according to the model identifier.

[0130] S3, determining whether the target neural network model is stored in the first storage space;

[0131] S4. If the target neural network model is not stored in the first storage space, pre-fetch the target neural network model from the second storage space and load it into the first storage space to cover at least one neural network model pre-stored in the first storage space;

[0132] The first storage space is the storage space required when the target neural network model is used to process the request, and the second storage space may be a storage space other than the first storage space. For example, when the network processor needs to use the target neural network model to process the request, the first storage space is the video memory, and correspondingly, the second storage space may be the memory.

[0133] S5. Processing the request based on the target neural network model stored in the first storage space;

[0134] S6. Send the processing result to the requesting party.

[0135] For example, Figure 4 As shown, if the sender is the user end, the receiver is the server end, the first storage space is the video memory, and the second storage space is the memory, then after the user end sends a request to the server end, the server end first determines that the target neural network model is model 1 based on the content of the request, and determines whether model 1 is stored in the video memory. Since only models 2-5 are stored in the video memory, model 1 is not stored in the video memory. At this time, model 1 can be pre-fetched from the memory storing models 1-6, and model 1 can be loaded into the video memory to cover model 5, so that the existing models in the video memory are models 1-model 4, and then model 1 is used to process the request, and finally the processing result is fed back to the user end.

[0136] Further, according to the above method embodiment, another embodiment of the present invention also provides a request processing device based on a neural network model, such as Figure 5 As shown, the device comprises:

[0137] A determination unit 41 is used to determine whether at least one target neural network model is stored in the video memory of the network processor, wherein the target neural network model includes a neural network model corresponding to the queue of pending requests;

[0138] A loading unit 42, configured to pre-fetch the target neural network model from the memory and load it into the video memory if the target neural network model is not stored in the video memory, so as to overwrite at least one neural network model pre-stored in the video memory;

[0139] The request processing unit 43 is used to process the requests in the pending request queue based on the target neural network model stored in the video memory.

[0140] Optional, such as Figure 6 As shown, the loading unit 42 includes:

[0141] A pre-fetching module 421, used for pre-fetching the target neural network model from the memory;

[0142] A determination module 422, used to determine whether the video memory contains a neural network model that is not executed;

[0143] The overwriting module 423 is used to make the target neural network model overwrite at least one neural network model in an unexecuted state based on the overwriting strategy if the video memory contains a neural network model in an unexecuted state.

[0144] Optionally, the judging unit 41 is further configured to judge whether the remaining space size of the video memory is greater than or equal to the size of the target neural network model if there is no neural network model in an unexecuted state in the video memory;

[0145] The loading unit 42 is also used to load the target neural network model directly into the video memory if the size of the remaining space is greater than or equal to the size of the target neural network model.

[0146] Optionally, the overwriting module 423 is used to randomly select a neural network model in an unexecuted state and use the target neural network model to overwrite the selected neural network model if the reserved space of the video memory is greater than a preset space threshold; if the reserved space of the video memory is less than the preset space threshold, use the target neural network model to overwrite at least one neural network model in an unexecuted state based on the size of the neural network model in the unexecuted state and the size of the target neural network model.

[0147] Optionally, the loading unit 42 is used to notify the network processor by a pre-fetch thread to pre-fetch the target neural network model from the memory and load it into the video memory to overwrite at least one neural network model pre-stored in the video memory.

[0148] Optional, such as Figure 6 As shown, the request processing unit 43 includes:

[0149] An adding module 431 is used for adding the name of the target neural network model to a pre-fetch task queue by a pre-fetch thread;

[0150] The notification processing module 432 is used for the pre-fetch thread to notify the working thread to process the request in the request queue corresponding to the name of the target neural network model in the pre-fetch task queue.

[0151] Optional, such as Figure 6 As shown, the device also includes:

[0152] A waiting processing unit 44 is used for, after the pre-fetch thread adds the name of the target neural network model to the pre-fetch task queue, if the number of names contained in the pre-fetch task queue is greater than or equal to a preset number threshold, the pre-fetch thread waits for the number of names contained in the pre-fetch task queue to decrease to less than the preset number threshold, and then processes the target neural network model corresponding to the next pending request queue;

[0153] The processing unit 45 is used to process the target neural network model corresponding to the next pending request queue if the number of names contained in the pre-fetch task queue is less than a preset number threshold.

[0154] Optionally, the judgment unit 41 is used to judge whether the target neural network model corresponding to the pending request queue is stored in the video memory if there is a request in the pending request queue.

[0155] The neural network model-based request processing device provided in an embodiment of the present invention can pre-store at least one neural network model in a memory whose storage space is much larger than the video memory. When it is necessary to use at least one target neural network model corresponding to the pending request queue to process the request, if the target neural network model is not stored in the video memory of the network processor, the target neural network model can be pre-fetched from the memory, and by overwriting at least one neural network model pre-stored in the video memory, the target neural network model can be successfully loaded into the video memory to process the request without causing the target neural network model to be unable to be loaded due to insufficient video memory space. Indirect expansion of the video memory is achieved through the memory, thereby realizing the function of loading large-scale neural network models using a single network processor, and can dynamically change rapidly with changes in traffic through software resources, thereby improving flexibility.

[0156] Further, according to the above method embodiment, another embodiment of the present invention also provides a request processing device based on a neural network model, such as Figure 7 As shown, the device comprises:

[0157] A determination unit 51, configured to determine whether at least one target language translation model is stored in the video memory of the graphics processor, wherein the target language translation model includes a neural network model for language translation corresponding to the pending request queue;

[0158] A loading unit 52, configured to pre-fetch the target language translation model from the memory and load it into the video memory if the target language translation model is not stored in the video memory, so as to overwrite at least one language translation model pre-stored in the video memory;

[0159] A request processing unit 53, configured to process the requests in the pending request queue based on the target language translation model stored in the video memory;

[0160] The output unit 54 is used to output the translation results corresponding to the requests in the pending request queue.

[0161] The neural network model-based request processing device provided in an embodiment of the present invention can pre-store at least one language translation model in a memory whose storage space is much larger than the video memory. When it is necessary to use at least one target language translation model corresponding to the pending request queue to process the request, if the target language translation model is not stored in the video memory of the network processor, the target language translation model can be pre-fetched from the memory, and by overwriting at least one language translation model pre-stored in the video memory, the target language translation model is successfully loaded into the video memory to process the request and output the translation result without causing the target translation model to be unable to be loaded due to insufficient video memory space, thereby indirectly expanding the video memory through the memory, and further realizing the function of loading large-scale language translation models using a single network processor, and can dynamically change rapidly with changes in traffic through software resources, thereby improving flexibility.

[0162] Further, according to the above method embodiment, another embodiment of the present invention also provides a request processing device based on a neural network model, such as Figure 8 As shown, the device comprises:

[0163] The receiving unit 61 is used to receive a request sent by a requesting party;

[0164] A determination unit 62, configured to determine, according to the request, at least one target neural network model required to process the request;

[0165] A judging unit 63, used to judge whether the target neural network model is stored in the first storage space;

[0166] A loading unit 64 is used for pre-fetching the target neural network model from the second storage space and loading it into the first storage space if the target neural network model is not stored in the first storage space, so as to overwrite at least one neural network model pre-stored in the first storage space;

[0167] A processing unit 65, configured to process the request based on the target neural network model stored in the first storage space;

[0168] The sending unit 66 is used to send the processing result to the requesting party.

[0169] Furthermore, another embodiment of the present invention also provides a storage medium, which stores multiple instructions, and the instructions are suitable for being loaded by a processor and executing the request processing method based on the neural network model as described above.

[0170] Furthermore, another embodiment of the present invention provides an electronic device, the electronic device comprising a storage medium and a processor;

[0171] The processor is adapted to implement each instruction;

[0172] The storage medium is suitable for storing a plurality of instructions;

[0173] The instructions are suitable for being loaded by the processor and executing the request processing method based on the neural network model as described above.

[0174] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0175] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific language is for disclosing an embodiment of the present invention.

[0178] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0179] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0180] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0181] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0182] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the request processing method and device based on a neural network model according to an embodiment of the present invention. The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0183] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

Claims

1. A request processing method based on a neural network model, characterized in that: The method comprises: Determining whether at least one target neural network model is stored in a video memory of a network processor, the target neural network model including a neural network model corresponding to a queue of pending requests; If the target neural network model is not stored in the video memory, pre-fetching the target neural network model from the memory; Determining whether the video memory contains a neural network model that is not executed; If the video memory contains a neural network model that is not executed, then based on the overwriting strategy, the target neural network model overwrites at least one neural network model that is not executed; The requests in the pending request queue are processed based on the target neural network model stored in the video memory.

2. The method according to claim 1, characterized in that The method further comprises: If there is no neural network model in an unexecuted state in the video memory, determining whether the remaining space size of the video memory is greater than or equal to the size of the target neural network model; If the size of the remaining space is greater than or equal to the size of the target neural network model, the target neural network model is directly loaded into the video memory.

3. The method according to claim 1, characterized in that Based on the coverage strategy, making the target neural network model cover at least one neural network model in an unexecuted state includes: If the reserved space of the video memory is greater than a preset space threshold, a neural network model that is not executed is randomly selected, and the target neural network model is used to overwrite the selected neural network model; If the reserved space of the video memory is less than a preset space threshold, the target neural network model is used to overwrite at least one of the neural network models in the unexecuted state based on the size of the neural network model in the unexecuted state and the size of the target neural network model.

4. The method according to claim 1, characterized in that: Pre-fetching the target neural network model from the memory and loading it into the video memory to cover at least one neural network model pre-stored in the video memory includes: The prefetch thread notifies the network processor to prefetch the target neural network model from the memory and load it into the video memory to overwrite at least one neural network model pre-stored in the video memory.

5. The method according to claim 1, characterized in that Processing the requests in the pending request queue based on the target neural network model stored in the video memory includes: The prefetch thread adds the name of the target neural network model to the prefetch task queue; The pre-fetch thread notifies the working thread to process the request in the request queue corresponding to the name of the target neural network model in the pre-fetch task queue.

6. The method according to claim 5, characterized in that After the prefetch thread adds the name of the target neural network model to the prefetch task queue, the method further includes: If the number of names contained in the pre-fetch task queue is greater than or equal to a preset number threshold, the pre-fetch thread waits for the number of names contained in the pre-fetch task queue to decrease to less than the preset number threshold, and then processes the target neural network model corresponding to the next pending request queue; If the number of names contained in the pre-fetch task queue is less than a preset number threshold, the pre-fetch thread processes the target neural network model corresponding to the next pending request queue.

7. The method according to any one of claims 1 to 6, characterized in that The determining whether at least one target neural network model is stored in the video memory of the network processor includes: If there are requests in the pending request queue, it is determined whether at least one target neural network model corresponding to the pending request queue is stored in the video memory.

8. A request processing method based on a neural network model, characterized in that: The method comprises: Determining whether at least one target language translation model is stored in a video memory of a graphics processor, the target language translation model comprising a neural network model for language translation corresponding to the pending request queue; If the target language translation model is not stored in the video memory, pre-fetching the target language translation model from the memory; Determining whether the video memory contains a language translation model that is not executed; If the video memory contains a language translation model in an unexecuted state, based on an overwriting strategy, the target language translation model overwrites at least one language translation model in an unexecuted state; Processing the requests in the pending request queue based on the target language translation model stored in the video memory; Output the translation results corresponding to the requests in the pending request queue.

9. A request processing method based on a neural network model, characterized in that: The method comprises: receiving a request sent by a requesting party; Determining, based on the request, at least one target neural network model required to process the request; Determining whether the target neural network model is stored in the first storage space; If the target neural network model is not stored in the first storage space, pre-fetching the target neural network model from the second storage space; Determining whether the first storage space contains a neural network model that is not executed; If the first storage space contains a neural network model in an unexecuted state, based on the coverage strategy, the target neural network model covers at least one neural network model in an unexecuted state; Processing the request based on the target neural network model stored in the first storage space; The processing result is sent to the requesting party.

10. A request processing device based on a neural network model, characterized in that: The device comprises: A determination unit, configured to determine whether at least one target neural network model is stored in a video memory of a network processor, wherein the target neural network model includes a neural network model corresponding to a queue of pending requests; A loading unit, configured to pre-fetch the target neural network model from the memory if the target neural network model is not stored in the video memory; determine whether the video memory contains a neural network model that is not executed; if the video memory contains a neural network model that is not executed, based on an overwriting strategy, enable the target neural network model to overwrite at least one neural network model that is not executed; A request processing unit is used to process the requests in the pending request queue based on the target neural network model stored in the video memory.

11. A request processing device based on a neural network model, characterized in that: The device comprises: A determination unit, configured to determine whether at least one target language translation model is stored in a video memory of a graphics processor, wherein the target language translation model includes a neural network model for language translation corresponding to the pending request queue; A loading unit, configured to pre-fetch the target language translation model from the memory if the target language translation model is not stored in the video memory; determine whether the video memory contains a language translation model in an unexecuted state; if the video memory contains a language translation model in an unexecuted state, based on an overwriting strategy, enable the target language translation model to overwrite at least one language translation model in an unexecuted state; a request processing unit, configured to process requests in the pending request queue based on the target language translation model stored in the video memory; The output unit is used to output the translation results corresponding to the requests in the pending request queue.

12. A request processing device based on a neural network model, characterized in that: The device comprises: A receiving unit, configured to receive a request sent by a requesting party; a determination unit, configured to determine, according to the request, at least one target neural network model required to process the request; A judging unit, used to judge whether the target neural network model is stored in the first storage space; A loading unit, configured to pre-fetch the target neural network model from the second storage space if the target neural network model is not stored in the first storage space; determine whether the first storage space contains a neural network model that is not executed; if the first storage space contains a neural network model that is not executed, based on a coverage strategy, enable the target neural network model to cover at least one neural network model that is not executed; a processing unit, configured to process the request based on the target neural network model stored in the first storage space; The sending unit is used to send the processing result to the requesting party.

13. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the request processing method based on a neural network model as described in any one of claims 1 to 9.

14. An electronic device, characterized in that: The electronic device includes a storage medium and a processor; The processor is adapted to implement each instruction; The storage medium is suitable for storing a plurality of instructions; The instructions are suitable for being loaded by the processor and executing the request processing method based on the neural network model as described in any one of claims 1 to 9.

Images