A service processing method, apparatus and device, and a storage medium

Abstract

The application discloses a service processing method and device, equipment and a storage medium. The method comprises the following steps: in the embodiment, each deep learning model is allocated at least one first process, all deep learning models are collectively allocated a second process, each deep learning model is loaded into at least one third process, the first process receives a request for calling a deep learning model, and service data in the request is transmitted to the second process; when the second process accumulates service data to meet a preset condition, all accumulated service data are combined into a batch of source data, and the source data is transmitted to the third process; and the third process calls the deep learning model to batch-process the source data, and obtains target data. The division of labor between the processes in the embodiment is clear, and the repeated occupation of resources is reduced. The deep learning model is called for batch processing through a parallel computing architecture, and the efficiency of operation can be significantly improved in the case of limited resources.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and more particularly to a business processing method, apparatus, device, and storage medium. Background Technology

[0002] With the continuous improvement of processors, memory and other resources and the continuous growth of data volume, deep learning models such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) have made great progress, and deep learning models are increasingly being used in many businesses such as security, autonomous driving, and speech synthesis.

[0003] Currently, each time a new deep learning model is deployed, a separate set of code is compiled for it to perform inference on business data. As the number of businesses increases, the number of deep learning models deployed also increases, resulting in greater resource consumption. Under resource constraints, the computational efficiency of deep learning models is relatively low. Summary of the Invention

[0004] This application provides a business processing method, apparatus, device, and storage medium to address the problem of improving the computational efficiency of deep learning models under resource constraints.

[0005] According to one aspect of this application, a business processing method is provided, wherein each deep learning model is assigned at least one first process, all the deep learning models are jointly assigned to a second process, and each deep learning model is loaded into at least one third process, the method comprising:

[0006] The first process receives a request to invoke the deep learning model and transmits the business data in the request to the second process;

[0007] When the second process accumulates the business data to meet the preset conditions, it merges all the accumulated business data into a batch of source data and transmits the source data to the third process.

[0008] The third process invokes the deep learning model to batch process the source data and obtain the target data.

[0009] According to another aspect of this application, a business processing apparatus is provided, the apparatus comprising at least one first process, a second process and at least one third process; each deep learning model is allocated at least one first process, all the deep learning models are shared in a second process, and each deep learning model is loaded into at least one third process;

[0010] The first process is used to receive a request to invoke the deep learning model and transmit the business data in the request to the second process;

[0011] The second process is used to merge all the accumulated business data into a batch of source data when the accumulated business data reaches the preset condition, and then transmit the source data to the third process;

[0012] The third process is used to call the deep learning model to batch process the source data and obtain the target data.

[0013] According to another aspect of this application, a service processing device is provided, the service processing device comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the business processing method described in any embodiment of this application.

[0017] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program for causing a processor to execute and implement the business processing method described in any embodiment of this application.

[0018] According to another aspect of this application, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the business processing method described in any embodiment of this application.

[0019] In this embodiment, each deep learning model is assigned at least one first process, all deep learning models share a second process, and each deep learning model is loaded into at least one third process. The first process receives requests to invoke the deep learning model and transmits the requested business data to the second process. When the second process accumulates business data to meet preset conditions, it merges all accumulated business data into a batch of source data and transmits the source data to the third process. The third process invokes the deep learning model to batch process the source data and obtain the target data. This embodiment uses a structure of first, second, and third processes to construct a scheduling system. The division of labor among the processes is clear, which not only reduces repetitive operations and thus reduces repetitive resource consumption, but also fully preserves the scalability of the scheduling system. By using a parallel computing architecture to invoke the deep learning model for batch processing, the efficiency of computation can be significantly improved under resource constraints.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a business processing method provided according to Embodiment 1 of this application;

[0023] Figure 2 This is an architecture diagram of a scheduling system provided according to Embodiment 1 of this application;

[0024] Figure 3 This is a flowchart of a business processing method provided according to Embodiment 2 of this application;

[0025] Figure 4 This is a schematic diagram of a business processing device according to Embodiment 3 of this application;

[0026] Figure 5 This is a schematic diagram of the structure of a business processing device provided in Embodiment 4 of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1 This is a flowchart illustrating a business processing method provided in Embodiment 1 of this application. This embodiment is applicable to situations where multiple deep learning models are scheduled under a unified architecture. The method can be executed by a business processing device, which can be implemented in hardware and / or software and can be configured within a business processing equipment. The business processing equipment is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The business processing equipment can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices.

[0031] Processors that support batch processing (i.e., parallel computing) can be configured in business processing devices, such as GPUs (Graphics Processing Units), NPUs (Neural-network Processing Units), DPUs (Data Processing Units), etc. In addition to processors that support batch processing, processors with general logic processing functions, such as CPUs (Central Processing Units), can also be configured in business processing devices.

[0032] In the business processing equipment, multiple deep learning models can be deployed according to business needs. The structure of the deep learning models is not limited to manually designed deep learning models, such as ResNet (residual network), LSTM (Long Short Term Memory), GAN (Generative Adversarial Networks), etc. It can also be a deep learning model optimized by model quantization methods, a deep learning model searched for business characteristics by NAS (Neural Architecture Search) methods, etc. This embodiment does not impose any restrictions on this.

[0033] In some designs, a deep learning model independently provides a complete business service, such as face recognition, TTS (Text to Speech), and so on.

[0034] In some designs, at least two deep learning models simultaneously provide a complete business service. For example, one deep learning model identifies traffic lights in image data, while another deep learning model identifies the countdown timer in the traffic lights, and so on.

[0035] Furthermore, for the same deep learning model, the services it provides can be a complete business service in some cases, and a part of a complete business service in others.

[0036] In this embodiment, in order to improve scheduling efficiency, one or more deep learning models that provide a complete business service can be divided into the same set for scheduling. That is, one or more deep learning models in the same set provide a complete business service.

[0037] In the same set, such as Figure 2As shown, each deep learning model is assigned at least one first process, all deep learning models are shared by one second process, and each deep learning model is loaded into at least one third process.

[0038] In this embodiment, the framework for scheduling multiple deep learning models can be compiled into a scheduling system (program). When the scheduling system starts, it creates a first process, a second process, a third process, and a fourth process respectively.

[0039] Among them, the first process, the second process, the third process, and the fourth process all belong to the category of processes. A program is a description of instructions, data, and their organization. A process is a running activity of a program on a certain data set and is the basic unit for resource allocation and scheduling. In other words, a process is an entity of a program.

[0040] When the second process starts, it creates a first thread, a second thread, and a third thread. The first thread, the second thread, and the third thread work together to complete the function of the second process.

[0041] In this context, the first thread, the second thread, and the third thread all belong to the category of threads. A thread is the smallest unit capable of execution scheduling. It is contained within a process and is the actual unit of operation within the process. A thread refers to a single sequential flow of control within a process. Multiple threads can run concurrently within a process (programs with many relatively independent execution flows share most of the program's data structures). Each thread executes different tasks in parallel, and threads belonging to the same process share all the resources owned by the process.

[0042] Within the same set, each deep learning model is assigned at least one first process, all deep learning models are shared in a second process, and each deep learning model is loaded into at least one third process.

[0043] The first process, also known as the data process, serves as the entry and exit point for the entire program. It handles business-related logic, receives business data, and feeds back the target data that the deep learning model infers based on the business data. Its initialization parameters can be configured, such as the name of the deep learning model, the path of the deep learning model, the maximum batch size, the input metadata (i.e., the first metadata), the output metadata (i.e., the second metadata), and so on.

[0044] The second process, also known as the backend process, serves as the entry and exit point for the deep learning model. It is responsible for scheduling and integrating business data into the deep learning model and the target data for deep learning model inference.

[0045] The third process, also known as the model process, is responsible for loading, starting, and calling the deep learning model to perform inference.

[0046] The fourth process, also known as the context process, is responsible for maintaining communication between the first and third processes in each set.

[0047] Furthermore, the first process communicates with the fourth process through a queue, and the second process communicates with the fourth process through a queue. Therefore, the first process communicates with the second process through the fourth process, and the first process communicates with the third process through shared memory, exchanging business data and target data.

[0048] When the business data and target data are in the format of image data, video data, audio data, etc., the size of the business data and target data is relatively large. If the business data and target data are exchanged using a queue, it will result in high latency. However, by exchanging the business data and target data themselves through shared memory, and transmitting address and metadata data through a queue, instead of directly exchanging the business data and target data themselves through the queue, the amount of data transmitted by the queue is small, which avoids affecting the transmission performance of the queue. This makes full use of shared memory, avoids the waste of resources caused by occupying memory without storing data, and reduces latency.

[0049] The first, second, and fourth processes can be deployed on processors with logical processing capabilities (such as CPUs), while the third process can be deployed on processors that support batch processing (such as GPUs) to improve the running efficiency of each process.

[0050] After the first process is created, it registers its own information with the fourth process, such as its ID, the name of the deep learning model it is responsible for, the path of the deep learning model it is responsible for, and so on.

[0051] like Figure 1 As shown, the method includes:

[0052] Step 101: The first process receives a request to invoke the deep learning model and transmits the business data in the request to the second process.

[0053] Depending on the business, the scheduling system can provide business services to users on the intranet or to users on the extranet. Therefore, the first process can receive requests from business modules located on the intranet and / or clients located on the extranet to invoke business services. These business services are provided by one or more deep learning models, which is equivalent to receiving a request to invoke one or more deep learning models.

[0054] In practical applications, the first and third processes allocated to a deep learning model are generally in a one-to-one correspondence. The number of first and third processes allocated to a deep learning model is positively correlated with the busyness of the business services that the deep learning model is responsible for. The number of first and third processes allocated to a deep learning model can be manually set by technical personnel according to the busyness of the business services, or the busyness can be represented in real time by indicators such as the resource utilization rate and call frequency of the deep learning model. Based on the busyness (indicators), the number of first and third processes allocated to the deep learning model can be dynamically adjusted. When busy, new first and third processes are dynamically allocated to the deep learning model, and when idle, the first and third processes already allocated to the deep learning model are released.

[0055] For some deep learning models with relatively idle business services, a first process and a third process can be assigned to them.

[0056] For some deep learning models with busy business services, at least two second processes and at least two third processes can be assigned to them.

[0057] If a deep learning model is assigned a first process, then that first process can directly receive requests to invoke the deep learning model.

[0058] If the same deep learning model is assigned to at least two first processes, then the at least two first processes can receive requests to call the deep learning model in a load-balanced manner.

[0059] In a load balancing approach, at least two first processes are ordered so that they can take turns receiving requests to invoke the deep learning model.

[0060] For example, a deep learning model provides content moderation services. Within the intranet, there's a business module responsible for content publishing logic. This deep learning model is assigned processes 1, 2, and 3. Client A publishes a short video, and the business module sends a content moderation request 1 to client A. Process 1 receives this request. Subsequently, client B publishes a short video, and the business module sends a content moderation request 2 to client B. Process 2 receives this request. Then, client C publishes a short video, and the business module sends a content moderation request 3 to client C. Process 3 receives this request. Then, client D publishes a short video, and the business module sends a content moderation request 4 to client D. Process 1 receives this request, and so on.

[0061] The request to invoke the deep learning model includes business data. For different business scenarios, business data is a data object that has its own characteristics in that business scenario.

[0062] Furthermore, the format of business data can include text data, image data, video data, audio data, point cloud data, and so on. The form in which business data is presented is related to the business scenario, and the same format of business data may be presented differently in different business scenarios.

[0063] Furthermore, the same business data can be in a single format or it can be a combination of multiple formats; this embodiment does not impose any restrictions on this.

[0064] For example, in the business scenario of autonomous driving, business data can be video data, audio data, and point cloud data collected by the vehicle from its surrounding environment. Point cloud data can be projected onto video data to infer the semantics of the surrounding environment. Audio data and video data can be used to infer the semantics of the surrounding environment separately or jointly, and so on.

[0065] For example, in the business scenario of photo entertainment, the business data can be photos (i.e., image data) taken by users using mobile terminals such as mobile phones, and the style transfer of these photos can be performed separately.

[0066] For example, in business scenarios such as e-book reading and short video production, business data can be novels or copywriting (i.e., text data), which can be converted into audio data separately.

[0067] So, if Figure 2 As shown, the first process can read business data from the request to call the deep learning model and transfer the business data to the second process.

[0068] In specific implementations, such as Figure 2 As shown, all sets (deep learning models) are shared by a fourth process. In the second process, at least one first thread is assigned to each deep learning model in the same set. The first thread loads the code in the second process that is responsible for the data sending and receiving logic, also known as the data receiving processor.

[0069] Generally, there is a one-to-one correspondence between the first thread and the first process. Therefore, the number of first threads is usually the same as the number of first processes.

[0070] The first process receives a request to invoke the deep learning model, writes the business data in the request into the shared memory established with the third process, and obtains the first address and the first metadata.

[0071] Each process (such as the first process, the third process, etc.) has its own Process Control Block (PCB) and Address Space, and each has a corresponding page table responsible for mapping the process's virtual addresses to physical addresses, managed by the Memory Management Unit (MMU). Two different virtual addresses are mapped to the same region of physical space through the page table; this region they point to is called shared memory.

[0072] Shared memory allows two unrelated processes (i.e., the first process and the third process) to access the same logical memory. Therefore, shared memory is a way for two running processes (i.e., the first process and the third process) to share and transfer data (such as business data and target data).

[0073] The first element information is data used to describe the business data stored in the first address, such as the directory tree structure, the mapping relationship between file, data block and copy storage locations, etc.

[0074] The first process writes the first address and the first metadata into the first queue established by the first process and the fourth process. The first queue is a bidirectional queue, meaning that both the first process and the fourth process can operate on it.

[0075] The fourth process reads the first address and the first metadata from the first queue and queries the second process allocated to the deep learning model.

[0076] If the second process is found, the fourth process writes the first address and the first metadata into the second queue established by the fourth process and the first thread in the second process. The second queue is a two-way queue, that is, a queue that can be operated by both the fourth process and the first thread in the second process.

[0077] If the second process is not found, the fourth process creates a second process for the first process and writes the first address and first metadata into the second queue established by the fourth process and the first thread in the second process.

[0078] This embodiment provides a fourth process to manage communication between the first and second processes, which can improve scheduling flexibility when deploying multiple deep learning models.

[0079] At this point, the first thread in the second process reads the first address and the first metadata from the second queue.

[0080] Step 102: When the second process accumulates business data to meet the preset conditions, it merges all the accumulated business data into a batch of source data and transmits the source data to the third process.

[0081] Under normal circumstances, such as Figure 2As shown, the second process faces multiple first processes. After receiving the business data transmitted by the first process, the second process does not immediately forward it to the third process. Instead, it caches and accumulates the business data. When the accumulated data meets the preset conditions, it merges all the accumulated business data into a batch of source data and transmits the source data to the third process.

[0082] For example, the condition may include the accumulated amount reaching a threshold, the accumulated time reaching a threshold, and so on.

[0083] In specific implementations, such as Figure 2 As shown, in the second process, a second thread is also allocated for all deep learning models in the same set. The second thread loads part of the code in the second process that is responsible for merging batch logic, also known as a data batch processor.

[0084] If the first process transmits the first address and first metadata of the business data to the second process, but not the business data itself, the first thread in the second process can configure an identifier for the business data it receives, thereby establishing a mapping relationship for a business data and transmitting the mapping relationship to the second thread.

[0085] In a mapping relationship, the first address, first metadata, and identifier of the same business data are included. The identifier is used to mark the unique information of the business data so as to distinguish the business data and the target data inferred by the deep learning model based on the business data in subsequent processing.

[0086] In general, in the second process, the second thread faces multiple first threads. When the second thread receives a mapping relationship transmitted by a first thread, it accumulates the mapping relationship. When the number of mapping relationships accumulates to a preset threshold, it packages all the currently accumulated mapping relationships into a batch of source data and writes the source data into the third queue established between the second process and the third process. At this time, the second thread starts to accumulate mapping relationships again.

[0087] The third queue is a bidirectional queue, meaning that it can be operated by both the second process and at least one third process.

[0088] At this point, each third process reads source data from the third queue, reads the mapping relationship it is responsible for (first address, first metadata and identifier) ​​from the source data, and reads the data itself of the business data from the first address of the shared memory according to the first metadata.

[0089] Step 103: The third process calls the deep learning model to batch process the source data and obtain the target data.

[0090] like Figure 2As shown, after receiving the source data, the third process uses the processor's parallel computing mechanism to call the deep learning model loaded into the third process in parallel to batch process the source data and obtain the target data.

[0091] Taking GPUs as an example of processors, parallel computing is a computing method that breaks down a specific computation into smaller, independent computations that can be performed simultaneously, and then recombines or synchronizes the results of the smaller computations to form the result of the original larger computation.

[0092] A larger task can be broken down into a number of tasks depending on the number of cores contained in a particular hardware. A core is the unit that actually performs computation in a given processor. CPUs typically have 4, 8, or 16 cores, while GPUs can typically have thousands, making GPUs suitable for parallel computing and completing tasks in parallel.

[0093] Deep learning models are embarrassingly parallel, making them suitable for parallel computation, and neural networks are also highly parallel. In parallel computing, a highly parallel task refers to a task that is divided into a set of smaller tasks for parallel computation; these smaller tasks are independent of each other. Because deep learning models are highly parallel, computations performed using deep learning models in many business scenarios can be decomposed into smaller computations, so that these smaller sets of computations do not depend on each other.

[0094] In one scenario, a set contains a single deep learning model, where each deep learning model independently provides a complete business service. In this case, each third process independently inputs business data into its loaded deep learning model for processing and outputs target data.

[0095] In another scenario, a set contains multiple deep learning models, where multiple deep learning models collaborate to provide a complete business service. In this case, each third process in the same set determines the dependencies between the multiple deep learning models, and sequentially calls the multiple deep learning models according to the dependencies to batch process the source data and obtain the target data.

[0096] In the dependency relationship, if the current deep learning model is depended on by other deep learning models, that is, other deep learning models depend on the current deep learning model, then the data output by the current deep learning model is the data input to other deep learning models. Initially, business data is input into a deep learning model that does not depend on any deep learning model for processing.

[0097] For example, in the business scenario of autonomous driving, two deep learning models simultaneously provide complete traffic light detection services. One deep learning model identifies traffic lights in image data, denoted as the traffic light recognition network, and the other deep learning model identifies the countdown in the traffic lights, denoted as the countdown recognition network. The traffic light recognition network is dependent on the countdown recognition network, that is, the countdown recognition network depends on the traffic light recognition network. So, initially, the image data is input to the traffic light recognition network for processing, and the output is the area where the traffic lights are located in the image data. The area where the traffic lights are located is input to the countdown recognition network for processing, and the output is the countdown in the traffic lights (i.e., the target data).

[0098] In this embodiment, each deep learning model is assigned at least one first process, all deep learning models share a second process, and each deep learning model is loaded into at least one third process. The first process receives requests to invoke the deep learning model and transmits the requested business data to the second process. When the second process accumulates business data to meet preset conditions, it merges all accumulated business data into a batch of source data and transmits the source data to the third process. The third process invokes the deep learning model to batch process the source data and obtain the target data. This embodiment uses a structure of first, second, and third processes to construct a scheduling system. The division of labor among the processes is clear, which not only reduces repetitive operations and thus reduces repetitive resource consumption, but also fully preserves the scalability of the scheduling system. By using a parallel computing architecture to invoke the deep learning model for batch processing, the efficiency of computation can be significantly improved under resource constraints.

[0099] Example 2

[0100] Figure 3 This is a flowchart of a business processing method provided in Embodiment 2 of this application. This embodiment adds a process for returning results based on the previous embodiment. Figure 3 As shown, the method includes:

[0101] Step 301: The first process receives a request to invoke the deep learning model and transmits the business data in the request to the second process.

[0102] Step 302: When the second process accumulates business data to meet the preset conditions, it merges all the accumulated business data into a batch of source data and transmits the source data to the third process.

[0103] Step 303: The third process calls the deep learning model to batch process the source data and obtain the target data.

[0104] Step 304: The third process transmits the target data to the second process.

[0105] like Figure 2As shown, when the deep learning model completes processing and outputs the target data, the third process transmits the target data to the second process.

[0106] In one scenario, a set contains a single deep learning model, where a single deep learning model independently provides a complete business service. In this case, each third process independently transmits the target data to the second process.

[0107] In another scenario, a set contains multiple deep learning models, where multiple deep learning models collaborate to provide a complete business service. In this case, each third process in the same set determines the dependencies between the multiple deep learning models, and the third process at the end of the dependency relationship transmits the target data to the second process.

[0108] In specific implementations, such as Figure 2 As shown, in the second process, a third thread is allocated to all deep learning models in the same set. The third thread loads the code in the second process that is responsible for the result processing logic, also known as the data result processor.

[0109] Then, the third process can write the target data into the shared memory established between the third process and the first process, and obtain the second address and the second metadata.

[0110] The first element information is data used to describe the target data stored in the second address, such as the directory tree structure, the mapping relationship between file, data block and copy storage locations, etc.

[0111] The third process queries the identifier corresponding to the business data cached locally, and writes the second address, second metadata, and identifier into the third queue established by the third process and the second process.

[0112] At this point, the third thread in the second process reads the second address, second metadata, and identifier from the third queue.

[0113] Step 305: The second process transmits the target data to the first process for output.

[0114] like Figure 2 As shown, the second process transmits the target data to the first process that provides business data. When the first process receives the target data, it can output the target data to the business module on the intranet or the client on the extranet.

[0115] In specific implementations, such as Figure 2 As shown, in the second process, at least one first thread is also assigned to each deep learning model in the same set.

[0116] The third thread in the second process transmits the second address and second metadata to the first thread according to the identifier. That is, the third thread queries the first thread that provides the identifier and transmits the second address and second metadata to the first thread. The first thread writes the second address and second metadata into the second queue established by the first thread and the fourth process in the second process.

[0117] At this point, the fourth process reads the second address and the second metadata from the second queue, and writes the second address and the second metadata into the first queue established by the fourth process and the first process.

[0118] The first process reads the second address and second metadata from the first queue, reads the target data from the second address in shared memory according to the second metadata, and outputs the target data to the business module on the intranet or the client on the external network.

[0119] In this embodiment, the third process transmits the target data for deep learning model inference to the second process, and the second process transmits the target data to the first process for output, ensuring the normal execution of the business. In this process, the scheduling division of labor between each process is clear, which can reduce repetitive operations, thereby reducing repetitive resource occupation and preserving the scalability of scheduling.

[0120] Example 3

[0121] Figure 4 This is a schematic diagram of a business processing device provided in Embodiment 3 of this application. Figure 3 As shown, the device includes at least one first process 401, one second process 402 and at least one third process 403; each deep learning model is assigned at least one first process, all the deep learning models are shared by one second process, and each deep learning model is loaded into at least one third process;

[0122] The first process 401 is used to receive a request to invoke the deep learning model and transmit the business data in the request to the second process 402;

[0123] The second process 402 is used to merge all the accumulated business data into a batch of source data when the accumulated business data meets the preset conditions, and then transmit the source data to the third process 403.

[0124] The third process 403 is used to call the deep learning model to batch process the source data and obtain the target data.

[0125] In one embodiment of this application, the apparatus further includes a fourth process shared by all the deep learning models, wherein at least one first thread is allocated to each of the deep learning models in the second process 402;

[0126] The first process 401 is further configured to receive a request to call the deep learning model, write the business data in the request into a shared memory established with the third process, obtain a first address and first metadata, and write the first address and first metadata into a first queue established with the fourth process;

[0127] The fourth process is used to read the first address and the first metadata from the first queue, query the second process allocated to the deep learning model, and write the first address and the first metadata into the second queue established with the first thread in the second process;

[0128] The first thread is used to read the first address and the first metadata from the second queue.

[0129] In one embodiment of this application, the first process 401 is further configured to receive requests to invoke the deep learning model in a load-balanced manner if the deep learning model is allocated at least two first processes.

[0130] In one embodiment of this application, a second thread is also allocated for all the said deep learning models in the second process 402;

[0131] The first thread is further configured to configure an identifier for the business data and transmit the mapping relationship to the second thread, wherein the mapping relationship includes the first address, the first metadata, and the identifier;

[0132] The second thread is used to package all the currently accumulated mapping relationships into a batch of source data when the number of the mapping relationships accumulates to a preset threshold, and write the source data into a third queue established with the third process;

[0133] The third process 403 is further configured to read the source data from the third queue and read the service data from the first address in the shared memory according to the first metadata.

[0134] In one embodiment of this application, each of the third processes 403 is further configured to independently input the business data into the deep learning model for processing and output target data;

[0135] or

[0136] Each of the third processes 403 is further configured to determine the dependencies between the multiple deep learning models, and sequentially call the multiple deep learning models according to the dependencies to batch process the source data to obtain the target data;

[0137] In the dependency relationship, if the current deep learning model is depended upon by other deep learning models, then the data output by the current deep learning model is the data input to the other deep learning models.

[0138] In one embodiment of this application, the third process 403 is further configured to transmit the target data to the second process 402;

[0139] The second process 402 is also used to transmit the target data to the first process for output.

[0140] In one embodiment of this application, a third thread is allocated for all the said deep learning models in the second process 402;

[0141] The third process 403 is further configured to write the target data into the shared memory established with the first process, obtain the second address and the second metadata, query the identifier corresponding to the business data, and write the second address, the second metadata and the identifier into the third queue established with the second process;

[0142] The third thread is used to read the second address, the second metadata, and the identifier from the third queue.

[0143] In one embodiment of this application, the apparatus further includes a fourth process shared by all the deep learning models, wherein the second process further allocates at least one first thread for each of the deep learning models;

[0144] The third thread is further configured to transmit the second address and the second metadata to the first thread according to the identifier;

[0145] The first thread is used to write the second address and the second metadata into the second queue established with the fourth process;

[0146] The fourth process is used to read the second address and the second metadata from the second queue, and write the second address and the second metadata into the first queue established with the first process;

[0147] The first process 401 is further configured to read the second address and the second metadata from the first queue, read the target data from the second address in the shared memory according to the second metadata, and output the target data.

[0148] The business processing apparatus provided in this application embodiment can execute the business processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the business processing method.

[0149] Example 4

[0150] Figure 5 A schematic diagram of the structure of a service processing device 10 that can be used to implement embodiments of this application is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of this application described and / or claimed herein.

[0151] like Figure 5 As shown, the service processing device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the service processing device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0152] Multiple components in the service processing device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network interface card, modem, wireless transceiver, etc. The communication unit 19 allows the service processing device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as business processing methods.

[0154] In some embodiments, the service processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the service processing device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the service processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the service processing method by any other suitable means (e.g., by means of firmware).

[0155] Example 5

[0156] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the business processing method provided in any embodiment of this application.

[0157] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A business processing method, characterized in that, Each deep learning model is assigned at least one first process, all deep learning models are shared in a second process, and each deep learning model is assigned at least one first thread within the second process. Each deep learning model is loaded into at least one third process, and all deep learning models are shared in a fourth process. The method includes: The first process receives a request to invoke the deep learning model and transmits the business data in the request to the second process; When the second process accumulates the business data to meet the preset conditions, it merges all the accumulated business data into a batch of source data and transmits the source data to the third process. The third process invokes the deep learning model to batch process the source data and obtain the target data; The first process receives a request to invoke the deep learning model and transmits the business data in the request to the second process, including: The first process receives a request to invoke the deep learning model, writes the business data in the request into a shared memory established with the third process, obtains a first address and first metadata, and writes the first address and first metadata into a first queue established with the fourth process. The fourth process reads the first address and the first metadata from the first queue, queries the second process allocated to the deep learning model, and writes the first address and the first metadata into the second queue established with the first thread in the second process; The first thread reads the first address and the first metadata from the second queue.

2. The method according to claim 1, characterized in that, The first process receives a request to invoke the deep learning model, including: If the deep learning model is allocated at least two first processes, the first processes receive requests to invoke the deep learning model in a load-balanced manner.

3. The method according to claim 1, characterized in that, The second process also allocates a second thread for all the deep learning models; when the second process accumulates the business data to meet a preset condition, it merges all the accumulated business data into a batch of source data and transmits the source data to the third process, including: The first thread configures an identifier for the business data and transmits the mapping relationship to the second thread. The mapping relationship includes the first address, the first metadata, and the identifier. When the number of the mapping relationships accumulates to a preset threshold, the second thread packages all the currently accumulated mapping relationships into a batch of source data and writes the source data into a third queue established with the third process. The third process reads the source data from the third queue and reads the service data from the first address in the shared memory according to the first metadata.

4. The method according to claim 3, characterized in that, Each of the aforementioned third processes invokes the deep learning model to batch process the source data and obtain the target data, including: Each of the aforementioned third processes independently inputs the business data into the deep learning model for processing and outputs the target data; or Each of the third processes determines the dependencies between the multiple deep learning models, and sequentially calls the multiple deep learning models according to the dependencies to batch process the source data to obtain the target data; In the dependency relationship, if the current deep learning model is depended upon by other deep learning models, then the data output by the current deep learning model is the data input to the other deep learning models.

5. The method according to any one of claims 1-4, characterized in that, Also includes: The third process transmits the target data to the second process; The second process transmits the target data to the first process for output.

6. The method according to claim 5, characterized in that, In the second process, a third thread is allocated for all the aforementioned deep learning models; The third process transmits the target data to the second process, including: The third process writes the target data into the shared memory established with the first process, obtains the second address and the second metadata, queries the identifier corresponding to the business data, and writes the second address, the second metadata, and the identifier into the third queue established with the second process. The third thread reads the second address, the second metadata, and the identifier from the third queue.

7. The method according to claim 6, characterized in that, All the deep learning models are shared in a fourth process, and in the second process, at least one first thread is also allocated to each of the deep learning models. The second process transmits the target data to the first process for output, including: The third thread transmits the second address and the second metadata to the first thread according to the identifier; The first thread writes the second address and the second metadata into the second queue established with the fourth process; The fourth process reads the second address and the second metadata from the second queue, and writes the second address and the second metadata into the first queue established with the first process; The first process reads the second address and the second metadata from the first queue, reads the target data from the second address in the shared memory according to the second metadata, and outputs the target data.

8. A business processing device, characterized in that, The device includes at least one first process, one second process, and at least one third process; each deep learning model is allocated at least one first process, all deep learning models are shared in one second process, each deep learning model is allocated at least one first thread in the second process, each deep learning model is loaded into at least one third process, and all deep learning models are shared in one fourth process. The first process is used to receive a request to invoke the deep learning model and transmit the business data in the request to the second process; The second process is used to merge all the accumulated business data into a batch of source data when the accumulated business data reaches the preset condition, and then transmit the source data to the third process; The third process is used to call the deep learning model to batch process the source data and obtain the target data; The first process is further configured to receive a request to invoke the deep learning model, write the business data in the request into a shared memory established with the third process, obtain a first address and first metadata, and write the first address and first metadata into a first queue established with the fourth process. The fourth process is used to read the first address and the first metadata from the first queue, query the second process allocated to the deep learning model, and write the first address and the first metadata into the second queue established with the first thread in the second process; The first thread is used to read the first address and the first metadata from the second queue.

9. A business processing device, characterized in that, The business processing equipment includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the business processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that causes a processor to execute the business processing method according to any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the business processing method according to any one of claims 1-7.

Images