Large model request processing method, word segmentation manager and equipment

Abstract

The invention discloses a request processing method of a large model, a word segmentation manager and equipment, and relates to the technical field of computers, the preset word segmentation manager is utilized to preprocess a request issued by a client, and an asynchronous calling method is utilized to call an inference engine to ensure that the large model performs inference and return on the request in time, so that the efficiency of the large model is improved. And furthermore, the receiving port responds to the returned data packet instantaneously, processes the data in time by utilizing an asynchronous event triggering mechanism and feeds back the data to the client, so that the data preprocessing efficiency of the large model is improved, and the performance of processing the request by the word segmentation manager is maximized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a large-model request processing method, a word segmentation manager, and a device. Background Art

[0002] With the rapid development of information technology, remarkable achievements have been made in the fields of artificial intelligence and natural language processing, especially in large language models.

[0003] As large models advance and market demand grows, users are increasingly using large model services, placing increasing pressure on the load. Improving the data processing efficiency of large models has become a pressing issue. Summary of the Invention

[0004] The present application provides a large model request processing method, word segmentation manager and device to at least solve the problem of low efficiency of large model preprocessing in related technologies.

[0005] This application provides a large-model request processing method, which is applied to the heterogeneous scheduling module of the word segmentation manager, including:

[0006] In response to the interaction port detecting the inference request sent by the client, detecting whether the event loop of the receiving port is created;

[0007] In response to detecting the event loop creation, triggering the word segmentation processing module to pre-process the inference request to generate a target request, create an asynchronous event, and cache the asynchronous event in a state queue;

[0008] Send the target request to the inference engine through the sending port to schedule the large model to process the target request and generate a request return data packet;

[0009] In response to detecting a request return data packet received by the receiving port, triggering the conjunction processing module to process the request return data packet to obtain an inference result and trigger an asynchronous event;

[0010] In response to an asynchronous event trigger, the trigger request output module outputs the inference result and transmits it to the client through the interactive port.

[0011] This application also provides a word segmentation management, including:

[0012] Asynchronous scheduling module, used to call functional modules and create asynchronous events. Functional modules include word segmentation processing module, word combination processing module, receiving port, sending port, interaction port and request output module;

[0013] Interaction port, used to receive requests from the client, including inference requests, information acquisition requests, and status configuration requests;

[0014] A word segmentation processing module, used to pre-process the inference request to generate a target request;

[0015] The word combination processing module is used to process the request return data packet to obtain the inference results and trigger asynchronous events;

[0016] The sending port is used to send the target request to the inference engine to schedule the large model to process the target request and generate a request return data packet;

[0017] Receiving port, used to receive request return data packets;

[0018] The request output module is used to output the inference results in response to asynchronous event triggering and transmit them to the client through the interactive port.

[0019] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned large model request processing methods when executing the computer program.

[0020] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned large model request processing methods are implemented.

[0021] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned large model request processing methods when executed by a processor.

[0022] Through the request processing method of the large model provided by this application, the pre-set word segmentation manager is used to pre-process the request sent by the client, and the inference engine is called using the asynchronous call method to ensure that the large model can return the reasoning to the request in a timely manner. The receiving port further responds to the returned data packet instantaneously and uses the asynchronous event trigger mechanism to process the data in a timely manner and feed it back to the client, thereby improving the efficiency of pre-processing data of the large model while maximizing the performance of the word segmentation manager in processing requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 A large model reasoning diagram provided as background technology for this application;

[0025] Figure 2A schematic diagram of a large model request processing method provided in an embodiment of the present application;

[0026] Figure 3 A service architecture diagram provided for an embodiment of the present application;

[0027] Figure 4 A schematic diagram of initializing a word segmentation manager provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of a request processing flow for a large model provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a word segmentation manager architecture provided in an embodiment of the present application;

[0030] Figure 7 A diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0033] As disclosed in the background technology, large models have been widely used in various fields. Large models are usually adjusted according to model type, hardware environment and deployment requirements, such as Figure 1 As shown in the figure, the process typically includes preprocessing input data, model loading, inference acceleration, result post-processing, and result output. During the preprocessing phase, raw data needs to be cleaned and formatted to convert the raw input into the format required by the large model for easy processing.

[0034] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. This application proposes a request processing method for the preprocessing of large models, which asynchronously schedules model requests issued by the client to maximize the efficiency of large model data preprocessing.

[0035] The embodiment of the present application provides a large model request method, which is applied to the word segmentation manager. The different degree scheduling module in the word segmentation manager performs the following steps, such as Figure 2 As shown, specifically including:

[0036] S1. In response to detecting an inference request sent by a client, detecting whether an event loop of the receiving port is created.

[0037] like Figure 3 As shown, in the present application, the word segmentation manager interacts with the client through an interactive port. In this embodiment, the interactive port is preferably set to the FastAPI interface. The FastAPI interface is a Python framework for building a modern and high-performance framework to achieve enrichment of functional integration. Of course, a simple API interface can also be used in some other implementation scenarios. This application does not limit the specific interface type. In the embodiment of the present application, the server type installed by the word segmentation manager is preferably set to Uvicorn. Uvicorn is a fast ASGI (Asynchronous Server Gateway Interface) server based on Python. It is often used in conjunction with asynchronous Python Web frameworks such as FastAPI and Starlette, and can efficiently handle high-concurrency requests. Therefore, the word segmentation manager disclosed in the embodiment of the present application uses Uvicorn to run the above-mentioned FastAPI application, which combines Uvicorn as a high-performance ASGI server and FastAPI's asynchronous programming characteristics based on Python. It can handle a large number of concurrent requests and has excellent performance. Under this design architecture, the word segmentation manager interface is simple, universal, and easy to expand.

[0038] When the word segmentation manager detects the issued inference request through the interactive port, the heterogeneous scheduling module immediately manages the inference request and calls other functional modules to process the request. Among them, the inference request includes but is not limited to a request to generate an answer, an encoding request, and an embedding request. The inference request requires request data preprocessing and needs to be sent to the inference engine to obtain the inference data returned by the large model. When the word segmentation manager receives the issued request, it can record the corresponding request issuance time, and then determine the request number based on the request issuance time; of course, the above request number can also be determined by other means, and this application does not limit this.

[0039] It is worth noting that if Figure 4 As shown, the word segmentation manager disclosed in this application needs to configure the FastAPI interface, server address and port according to the functions supported by the manager before running; then start the inference engine, use the inference engine to load the large model and schedule the graphics processor to complete the calculation task of the large model; then initialize the word segmentation manager, preheat the server, start the server, and start listening for requests from the client.

[0040] Among them, in order to ensure the normal operation of the word segmentation manager and avoid abnormal processing of client requests due to word segmentation manager anomalies, this application proposes that after the word segmentation manager is initialized and before it is put into formal use, that is, before the inference request issued by the client, the word segmentation manager performs a self-test on the word segmentation manager: receiving a simulated client request, generating a simulated inference result that matches the client request and feeding it back to the client; receiving a test result generated by the client based on the simulated inference result and the expected inference result, the test result includes a qualified result and a failed result. If the similarity between the simulated inference result and the expected inference result is greater than a preset ratio, the test result is determined to be a qualified result, otherwise the test result is determined to be a failed result; in response to the test result being a qualified result, the client request monitoring function is started; in response to the test result being a failed result, a repair operation is performed, and of course an alarm prompt can also be triggered to prompt the staff to perform maintenance. Among them, the expected inference result can be the inference result successfully generated by the successfully trained large model for the inference request. The above preset ratio is preferably set to 90%, of course, it can also be set by those skilled in the art according to actual conditions, and this application does not limit this. Through the above self-test setting of the word segmentation manager, the reliability of the word segmentation manager is greatly improved.

[0041] In a specific implementation scenario, the above-mentioned detection of whether the event loop of the receiving port is created includes detecting whether the event loop flag is a preset value, wherein the preset value is a numerical value representing a Boolean value type, and the above-mentioned preset value is true; if the event loop flag is the preset value, it is determined that the event loop of the receiving port is created; if the event loop flag is a non-preset value, it is determined that the event loop of the receiving port is not created. Preferably, the above steps can be performed by an asynchronous scheduling module. Of course, a new logical unit can also be added to the word segmentation manager to perform the above steps, and this application does not limit this.

[0042] It is worth noting that the event loop created for the receiving port is used to detect whether the receiving port has return data. Every time the word segmentation manager detects a request sent by the client, it needs to query whether the event loop of the receiving port has been created to ensure that the receiving port service has been started, so as to ensure that the receiving port can receive the subsequent external feedback data of each request, further improve the stability of the word segmentation manager, and improve the robustness of the overall large model processing process.

[0043] Of course, after detecting whether the event loop of the receiving port is created, the word segmentation manager is also used to obtain the programming language of the word segmentation manager in response to detecting that the event loop has not been created; and call the preset script that matches the programming language to create the event loop. The preset script is a pre-written script for creating an event loop written in a variety of programming languages; matching scripts can be found according to the programming language of the word segmentation manager to achieve the creation of event loops in different types of word segmentation managers, thereby improving the adaptability of the word segmentation manager. In addition, it is also necessary to create asynchronous tasks at the receiving port, including service tasks and watchdog tasks. The service task is used to receive and process the received data, and the watchdog task is used to exit when an exception is received. Among them, the asynchronous task only needs to be created when the word segmentation manager receives the first request, and other requests can be obtained without having to be created again.

[0044] S2. In response to detecting the creation of an event loop, trigger the word segmentation processing module to pre-process the inference request to generate a target request and create an asynchronous event.

[0045] When triggering the word segmentation processing module to preprocess an inference request, the number of calls to the word segmentation processing module is determined based on the number of samples contained in the inference request. Specifically, the number of samples contained in the inference request, i.e., the batch, is obtained. If the number of samples is 1, the heterogeneous scheduling module calls the word segmentation processing module once to segment the inference request to generate the target request. If the number of samples is greater than 1, the word segmentation processing module is called a specified number of times to segment the inference request to generate the target request, where the specified number matches the number of samples. The word segmentation processing module preprocesses the inference request.

[0046] It is worth noting that in the embodiment of the present application, before triggering the word segmentation processing module to pre-process the inference request to generate the target request and create the asynchronous event, it is also proposed to: detect whether the file resource accessed by the inference request belongs to a shared resource; in response to detecting that the file resource belongs to a shared resource, obtain the exclusive lock of the file resource and then process the data packet returned by the request; in response to detecting that the file resource does not belong to a shared resource, process the data packet returned by the request. That is, during the operation of the large model, the word segmentation manager is set as the main process, the inference engine is located in the child process, and an exclusive lock is configured for the word segmentation manager. When it is necessary to process the shared data between processes, it is first determined whether the data is a shared resource. If so, the file resource is exclusively acquired based on the exclusive lock pre-set for the word segmentation manager, and then the data packet is processed to write the data in the data packet into the file resource. This greatly ensures the security of data reading and writing in the word segmentation manager, while ensuring the independent operation of each process during the large model inference process, further improving the stability of the data pre-processing of the entire large model.

[0047] Of course, in some implementation scenarios, before triggering the word segmentation processing module to pre-process the inference request to generate the target request and create an asynchronous event, the request needs to be initialized, such as input adoption number verification, standardization, sample size calculation, and default parameter filling, etc., to facilitate the transmission of the request in word segmentation management.

[0048] When the word segmentation manager pre-processes the inference request issued by the client to generate a target request, it needs to set an asynchronous event and cache the asynchronous event in the manager's request status queue, indexed by the request number. Furthermore, to facilitate the subsequent receiving port to receive the data returned by the large model for the target request, a request status data packet needs to be set at the same time as the asynchronous event is created. The request status data packet is also cached in the request status queue, indexed by the request number. It is worth noting that the data packet format is set by those skilled in the art based on actual needs. For example, as shown in Table 1, it consists of information such as the return data list of the target request, the inference status list (the inference status can be represented by a Boolean value), the asynchronous event, the request time, and the request return data packet. Of course, attributes such as the first data time and output offset can also be added to the data packet format, and the specific setting is determined by those skilled in the art based on actual conditions. The receiving port fills the request status data packet with data based on the request return data packet received from the inference engine. This application designs a method for easily managing and setting the request status data packet for inference requests, making it easy to manage information such as the request input, output, output delay, and inference status, facilitating targeted data processing.

[0049] Table 1

[0050]

[0051] S3. Send the target request to the inference engine through the sending port to schedule the large model to process the target request and generate a request return data packet.

[0052] For the target request generated based on the inference request, the heterogeneous scheduling module will call the sending port to send the target request to the inference engine, thereby obtaining the request return data packet generated by the large model for the target request containing data such as the inference result. It can be understood that in this application, the sending port and the receiving port are both inter-process communication interfaces created using ZeroMQ (Message Queue) for communication between the word segmentation manager and the inference engine. ZeroMQ, also known as 0MQ or ZMQ, is a high-performance asynchronous message library that provides the function of a message queue, but unlike traditional message queue software, it is more like an embedded network library.

[0053] S4. In response to detecting the request return data packet received by the receiving port, trigger the conjunction processing module to process the request return data packet to obtain an inference result and trigger an asynchronous event.

[0054] It should be noted that after the sending port sends the target request to the inference engine, the word segmentation manager is in a state of waiting for the request's asynchronous event to be triggered. When the receiving port receives the request return data packet, it obtains the corresponding request status based on the request number. The asynchronous scheduling module calls the corresponding word combination processing module to process the data based on the data type contained in the returned request return data packet, triggers the corresponding asynchronous event, and updates the data, request termination reason, and event status to the request status queue. The above-mentioned word combination processing module is the reverse process of word segmentation processing, which is used to reassemble the segmented word units into the original text.

[0055] The type of returned data corresponds to the data in the request status data packet specified in Table 1. Specifically, for inference requests, after receiving the returned request return data packets, the word segmentation manager needs to further process these requests. This means triggering the word combination processing module to process the returned request return data packets (i.e., word combination processing). An asynchronous event is triggered after detecting that word combination processing is complete.

[0056] S5. In response to the asynchronous event triggering, the trigger request output module outputs the inference result to the client.

[0057] In this application, after the asynchronous event is triggered, the asynchronous scheduling module performs the next step of processing for the inference request, that is, outputs the inference result.

[0058] like Figure 5As shown in the flowchart, after the word segmentation manager is initialized, the application starts the word segmentation manager and listens to whether the client sends a request, and then determines whether the event loop of the receiving port is created. The heterogeneous scheduling module performs corresponding scheduling processing according to the type of request. If the request is an inference request, the request is pre-processed after obtaining the thread exclusive lock, and then an asynchronous event and request status data packet are created and cached in the status queue, and then the pre-processed request is sent to the inference engine; wait for the state change, that is, wait for the receiving port to receive the returned request return data packet, further determine the data packet type and process the data packet according to the corresponding request status of the data packet, and then update the request status queue, and trigger an asynchronous event to output the inference result to the client using the interactive interface. The word segmentation manager is placed in the main thread, and the request is pre-processed first. The inference engine is called using the asynchronous call method to ensure that the large model returns the inference to the request in a timely manner. The receiving port further responds to the returned data packet instantaneously and uses the asynchronous event trigger mechanism to process the data in a timely manner and feeds it back to the client, thereby improving the efficiency of pre-processing data of the large model while maximizing the performance of the word segmentation manager in processing requests.

[0059] It is understandable that during the operation of a large model, the requests sent by the client include not only inference requests, but also information acquisition requests and status configuration requests. This application also proposes corresponding request processing methods for information acquisition requests and status configuration requests.

[0060] In some implementation scenarios, the word segmentation manager detects an information acquisition request issued by the client in response to the interactive port, including but not limited to a model information acquisition request and an inference engine information acquisition request. These requests only need to return the word segmentation management variable information, and there is no need to send a request to the inference engine. Then, it detects whether the event loop of the receiving port is created; in response to detecting that the event loop of the receiving port has been created, the heterogeneous scheduling module obtains the target information that matches the information acquisition request and triggers the request output module to output the target information to the client. It is specifically achieved by parsing the information acquisition request, and the specific steps are conventional technical means in this field, and this application will not be expanded here. It can be understood that in the scenario of responding to the information acquisition request, there is no need to call a large model, there is no inference data, and there is no need to create a request status data packet. The word segmentation manager can directly feed back the information to the client, that is, directly call the sending port and accept the occurrence status and return information.

[0061] In some implementation scenarios, the word segmentation manager detects a status configuration request issued by the client in response to the interactive port, including but not limited to a memory operation request, a performance detection request, and other configuration requests, wherein the memory operation request includes but is not limited to a cache refresh request, a request to update or initialize a single-machine memory weight, a request to update or initialize a distributed memory weight, and a request to release or cache memory occupancy. In response to detecting the creation of an event loop, the manager configuration is updated according to the status configuration request and the request output module is triggered to output the update information to the client. That is, these requests do not require request data preprocessing, do not need to return inference data, only need to return a success or failure status, and do not require further processing. The request status (success or failure) can be obtained by designing a request command publisher in the word segmentation manager, and then output through the request output module.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0063] The embodiment of the present application also provides a word segmentation manager, such as Figure 6 As shown, it includes an asynchronous scheduling module 610 for calling function modules and creating asynchronous events. The function modules include a word segmentation processing module 620, a word combination processing module 630, a receiving port 640, a sending port 650, an interaction port 660, and a request output module 670;

[0064] The word segmentation processing module 620 is used to pre-process the inference request to generate a target request;

[0065] The word combination processing module 630 is used to process the request return data packet to obtain the inference result and trigger the asynchronous event;

[0066] Receiving port 640, used to receive request return data packets;

[0067] Sending port 650 is used to send the target request to the inference engine to schedule the large model to process the target request and generate a request return data packet;

[0068] Interaction port 660 is used to receive requests from the client, including inference requests, information acquisition requests, and state configuration requests;

[0069] The request output module 670 is used to output the inference result in response to the asynchronous event trigger and transmit it to the client through the interactive port.

[0070] In some implementation scenarios, the request output module includes a single-request output module and a multi-request output module, which performs adaptive scheduling according to the number of samples included in the request.

[0071] For the description of the features in the embodiment corresponding to the word segmentation manager, please refer to the relevant description of the embodiment corresponding to the request processing method of the large model, which will not be repeated here.

[0072] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-mentioned large model request processing method embodiments:

[0073] In response to the interaction port detecting the inference request sent by the client, detecting whether the event loop of the receiving port is created;

[0074] In response to detecting the event loop creation, triggering the word segmentation processing module to pre-process the inference request to generate a target request, create an asynchronous event, and cache the asynchronous event in a state queue;

[0075] Send the target request to the inference engine through the sending port to schedule the large model to process the target request and generate a request return data packet;

[0076] In response to detecting a request return data packet received by the receiving port, triggering the conjunction processing module to process the request return data packet to obtain an inference result and trigger an asynchronous event;

[0077] In response to an asynchronous event trigger, the trigger request output module outputs the inference result and transmits it to the client through the interactive port.

[0078] in, Figure 7 The electronic device architecture is shown as an example, and may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a bus 730.

[0079] Among them, the processor 710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in this application.

[0080] The memory 720 can be implemented in the form of ROM (Read Only Memory, programmable memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the execution of the electronic device 700, and a basic input and output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and an icon font processing system 725, etc. can also be stored. The above-mentioned icon font processing system 725 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0081] The input / output interface 713 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0082] The network interface 714 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0083] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710 , the video display adapter 711 , the disk drive 712 , the input / output interface 713 , the network interface 714 , and the memory 720 ).

[0084] In addition, the electronic device 700 can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment, etc.

[0085] It should be noted that although the above device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal execution. In addition, those skilled in the art will understand that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0086] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned large model request processing method embodiments when running.

[0087] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0088] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned large model request processing method embodiments are implemented.

[0089] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned large model request processing method embodiments.

[0090] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] The above is a detailed introduction to the request processing method and word segmentation manager of a large model provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of the claims of this application.

Claims

1. A large model request processing method, characterized in that: An asynchronous scheduling module applied to a word segmentation manager, the method comprising: In response to the interaction port detecting the inference request sent by the client, detecting whether the event loop of the receiving port is created; In response to detecting the event loop creation, triggering the word segmentation processing module to pre-process the inference request to generate a target request, create an asynchronous event, and cache the asynchronous event in a state queue; Send the target request to the inference engine through the sending port to schedule the large model to process the target request and generate a request return data packet; In response to detecting the request return data packet received by the receiving port, triggering the conjunction processing module to perform data processing on the request return data packet to obtain an inference result and trigger an asynchronous event; In response to an asynchronous event trigger, the trigger request output module outputs the inference result and transmits it to the client through the interactive port.

2. The method according to claim 1, characterized in that The trigger word segmentation processing module pre-processes the inference request to generate a target request, including: Obtaining the number of samples included in the inference request; If the number of samples is 1, the word segmentation processing module is called once to segment the inference request to generate a target request; If the number of samples is greater than 1, the word segmentation processing module is called a specified number of times to segment the inference request to generate a target request, wherein the specified number of times matches the number of samples.

3. The method according to claim 2, characterized in that Before the triggering word segmentation processing module pre-processes the inference request to generate a target request and triggers the asynchronous scheduling module to create an asynchronous event, the method further includes: Detect whether the file resource accessed by the inference request is a shared resource; In response to detecting that the file resource is a shared resource, acquiring an exclusive lock of the file resource and then processing the request return data packet; In response to detecting that the file resource does not belong to a shared resource, data processing is performed on the request return data packet.

4. The method according to claim 1, wherein The method further comprises: In response to the interactive port detecting an information acquisition request sent by the client, detecting whether an event loop of the receiving port is created; In response to detecting the event loop creation, target information matching the information acquisition request is acquired and a request output module is triggered to output the target information to the client.

5. The method according to claim 1, wherein The method further comprises: In response to checking the status configuration request sent by the client, detecting whether the event loop of the receiving port is created; In response to detecting the event loop creation, the manager configuration is updated according to the state configuration request and a request output module is triggered to output update information to the client.

6. The method according to any one of claims 1 to 5, characterized in that The detecting whether the event loop of the receiving port is created includes: Check if the event loop flag is the preset value; If the event loop identifier is the preset value, determining the event loop creation of the receiving port; If the event loop identifier is a non-preset value, it is determined that the event loop of the receiving port is not created.

7. The method according to claim 6, characterized in that After detecting whether the event loop of the receiving port is created, the method includes: In response to detecting that the event loop is not created, obtaining a programming language of the word segmentation manager; A preset script matching the programming language is called to create the event loop.

8. A word segmentation manager, characterized in that: The word segmentation manager includes: An asynchronous scheduling module is used to call the functional modules and create asynchronous events. The functional modules include a word segmentation processing module, a word combination processing module, a receiving port, a sending port, an interaction port, and a request output module; The interactive port is used to receive requests sent by the client, including inference requests, information acquisition requests, and state configuration requests; The word segmentation processing module is used to pre-process the inference request to generate a target request; The word combination processing module is used to process the request return data packet to obtain an inference result and trigger an asynchronous event; The sending port is used to send the target request to the inference engine to schedule the large model to process the target request and generate a request return data packet; The receiving port is used to receive a request return data packet; The request output module is used to output the inference result in response to the asynchronous event triggering and transmit it to the client through the interactive port.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the large model request processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the large model request processing method according to any one of claims 1 to 7 are implemented.

Images