Computing task processing method and system for ai inference service

Abstract

The application discloses a kind of computing task processing methods and systems for AI inference service, wherein the method comprises: front-end server receives the file to be processed uploaded by client, and is written into the local storage path of inference server by network sharing protocol;Wherein, the local storage path of inference server is pre-mapped as the local mounting path of front-end server;Front-end server generates task message containing local storage path after the file to be processed is written successfully, and publishes task message to message middleware;Inference server obtains task message from message middleware, and reads the file to be processed according to the local storage path in task message through local file system interface.The above-mentioned method realizes point-to-point direct from data flow from access end to computing end, completely eliminates the network I / O load of inference server from external storage remote pulling file, significantly improves the overall throughput and hardware investment return rate of system.

Description

Technical Field

[0001] This invention relates to the field of network computing task processing technology, and in particular to a computing task processing method and system for AI inference services. Background Technology

[0002] With the popularization of artificial intelligence (AI) technology and the continuous expansion of its application fields, deploying AI models and providing online inference services (such as image recognition and video content moderation) has become an important application scenario in the current field of computer system architecture and high-performance computing. These systems typically face extremely high concurrent client access volumes and massive multimedia file upload demands, placing extremely high demands on system throughput, response latency, and the utilization rate of underlying computing resources (especially GPU resources).

[0003] Early AI model inference service systems typically employed a single, integrated server architecture. In this architecture, a high-performance server equipped with a GPU was required to handle all tasks end-to-end, including: processing high-concurrency network file upload requests (involving significant CPU consumption for network I / O and protocol parsing), saving files to local disks, utilizing the GPU for intensive matrix operations to complete model inference, and finally returning the results to the client. However, with the increasing number of AI service connections, this highly coupled architecture revealed significant limitations: when faced with sudden surges in concurrent file uploads, pre-processing tasks such as network I / O and file storage would fiercely compete for CPU resources, causing CPU utilization to reach saturation before GPU utilization. At this point, the entire processing pipeline was blocked in the data preparation phase, leaving expensive GPU computing resources idle or underutilized, creating a resource mismatch phenomenon of "CPU bottleneck, GPU waiting," severely limiting the overall system throughput.

[0004] To address the aforementioned issues, those skilled in the art have proposed a decoupled cluster architecture based on load balancing and centralized shared storage. This solution typically includes a front-end application server cluster, a centralized shared storage system (such as NAS or object storage), a message queue, and a dedicated GPU computing cluster. The workflow is as follows: the client uploads a file to the front-end application server; the front-end application server "pushes" the file to the dedicated centralized shared storage system; subsequently, a task description containing the storage path is generated and sent to the message queue; finally, the servers in the GPU computing cluster retrieve the task from the message queue and actively "pull" the file to be analyzed from the centralized shared storage system via the network for inference computation.

[0005] Although the above cluster architecture achieves a preliminary decoupling of access distribution and computing tasks at the physical node level, this solution still has the following inherent technical defects in actual high-concurrency engineering practices: First, in this architecture, the GPU server still needs to handle I / O operations for remotely reading (fetching) files from a separate centralized shared storage. The file reading process involves CPU-intensive tasks such as network packet processing and storage protocol parsing. Therefore, the CPU resources of the GPU server will still be occupied by network data transfer work, preventing the computing units from focusing 100% on core model inference, resulting in the GPU computing power not reaching its theoretical peak.

[0006] Second, the existing architecture relies on a separate centralized storage as the data convergence point for all nodes. All data streams for uploading, writing, reading, and analysis must pass through this central hub, resulting in a lengthy data flow path (client → front-end server → central storage → GPU server). In high-concurrency scenarios, the network bandwidth and disk IOPS of the central storage can easily become bottlenecks for the overall system throughput, while also introducing higher network latency and single-point-of-failure risks.

[0007] In summary, while existing AI inference service architectures achieve logical decoupling of task flows, they fail to achieve optimal configuration of data flows and underlying hardware resources, and cannot completely eliminate the I / O burden on GPU servers. Therefore, a new method for processing computational tasks is urgently needed. Summary of the Invention

[0008] The purpose of this invention is to provide a computational task processing method and system for AI inference services that can completely eliminate the I / O burden of the inference server.

[0009] To achieve the above objectives, the present invention provides a computational task processing method for AI inference services, comprising: The front-end server receives the file to be processed uploaded by the client and writes the file to be processed into the local storage path of the inference server through a network sharing protocol; wherein, the local storage path of the inference server is pre-mapped to the local mount path of the front-end server. After the file to be processed is successfully written, the front-end server generates a task message containing the local storage path and publishes the task message to the message middleware. The inference server obtains the task message from the message middleware and reads the file to be processed through the local file system interface according to the local storage path in the task message; The inference server performs AI model inference processing on the read file to be processed, and deletes the corresponding file to be processed in the local storage path after the processing is completed.

[0010] Preferably, the method by which the front-end server receives the file to be processed uploaded by the client includes: The front-end server receives file upload requests initiated by the client through the front-end load balancing layer; The front-end load balancing layer distributes the upload request to any one of the application server clusters with multiple front-end servers according to a preset strategy.

[0011] Preferably, the network sharing protocol is the Network File System (NFS) protocol; The inference server acts as an NFS server, exporting its local specified directory to the local storage path; The front-end server, acting as an NFS client, mounts the exported specified directory to its own local file system, generating the local mount path.

[0012] Preferably, the local storage path in the task message is the local absolute path of the file to be processed on the inference server; The front-end server publishes the task message to a designated switch of the message middleware and binds it to a dedicated task queue for the inference server to listen to via a routing key.

[0013] Preferably, the inference server is equipped with a graphics processing unit (GPU); The inference server transmits the data path of the file to be processed to the underlying AI model through a preset interface, and drives the GPU to perform inference calculations on the file to be processed to generate inference results.

[0014] Preferably, after generating the inference result, the inference server saves or sends the inference result to the result notification queue; After confirming that the inference result processing is complete, the analysis process of the inference server executes a file deletion command to clear the files to be processed under the local storage path, thereby freeing up the local storage space of the inference server.

[0015] Preferably, before the front-end server writes the file to be processed to the local storage path of the inference server via a network sharing protocol, it further includes a capacity detection and overload prevention method: The front-end server detects the current remaining storage capacity of the storage volume where the local mount path is located; If the current remaining storage capacity is lower than a preset security threshold, the front-end server will suspend writing the file to be processed to the inference server and trigger a traffic scheduling policy to route the file to be processed to the local storage path of another standby inference server in normal capacity, or return a rate-limiting retry response to the client.

[0016] The present invention also provides a computational task processing system for AI inference services, which includes a front-end server and an inference server, wherein the front-end server and the inference server process user-uploaded files based on the computational task processing method described above.

[0017] The present invention also provides a computational task processing system for AI inference services, comprising: One or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for performing the computational task processing method as described above.

[0018] The present invention also provides a computer-readable storage medium comprising a computer program that can be executed by a processor to perform the computational task processing method described above.

[0019] Compared to existing technologies, the computational task processing method provided by the above technical solution maps the local storage of the inference server to the mount path of the front-end server, realizing point-to-point direct data flow from the access end to the computing end, completely eliminating the network I / O load of the inference server remotely pulling files from external storage. This architecture allows the CPU of the inference server to be freed from heavy network protocol parsing and data transfer tasks, and can focus entirely on driving the computing acceleration unit to perform model inference, fundamentally solving the resource mismatch problem of CPU bottleneck leading to idle computing power of the inference server in high-concurrency scenarios. At the same time, the extremely simplified data path significantly reduces processing latency, and together with the closed-loop mechanism of automatically cleaning up files after inference, it effectively eliminates the bandwidth bottleneck and single point of failure risk of centralized storage, significantly improving the overall throughput and return on hardware investment while simplifying the complexity of operation and maintenance. Attached Figure Description

[0020] Figure 1 This is a diagram of the computing task processing system architecture in an embodiment of the present invention.

[0021] Figure 2 This is a flowchart of the computational task processing method in an embodiment of the present invention. Detailed Implementation

[0022] To illustrate the technical content, structural features, objectives, and effects of the present invention in detail, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0023] This embodiment discloses a computational task processing method for AI inference services, applicable to online service scenarios such as image recognition, video content review, speech recognition, and document parsing, which require clients to upload files and AI models to perform inference processing. The system upon which this method relies includes a front-end server 100, an inference server 200, and a message middleware 300. The front-end server 100 receives files uploaded by clients, the inference server 200 executes AI model inference processing, and the message middleware 300 transmits task messages between the front-end server 100 and the inference server 200.

[0024] In this embodiment, the inference server 200 has a local storage path located within its own file system. This local storage path can be, for example, a local disk, a solid-state drive, or a directory mounted on a storage volume on the inference server 200 itself. This local storage path is pre-mapped to the local mount path of the front-end server 100 via a network sharing protocol. In other words, for the front-end server 100, this local mount path appears as a directory within its local file system.

[0025] Therefore, please refer to the following: Figure 1 and Figure 2 The computational task processing method in this embodiment includes the following steps: S1: Front-end server 100 receives the file to be processed uploaded by the client and writes the file to be processed into the local storage path of inference server 200 through network sharing protocol.

[0026] The client can send an upload request to the front-end server 100 via HTTP, HTTPS, or other application layer protocols. After receiving the upload request, the front-end server 100 reads the file stream of the file to be processed from the request body and generates a filename for the file to be processed.

[0027] The front-end server 100 does not need to save the files to be processed to its own local disk before forwarding them. Instead, it writes the received file stream directly to the target file under the local mount path.

[0028] Because a mapping relationship has been established between the local mount path and the local storage path of the inference server 200 through a network sharing protocol, file write operations performed by the front-end server 100 on the local mount path are converted by the network sharing protocol into write operations on the local storage path of the inference server 200. After the file to be processed is written, the corresponding entity file exists in the local storage path of the inference server 200. For example, when the front-end server 100 writes a file with the path " / mnt / ai_share / uuid1234.mp4", the file " / AI_FILE / uuid1234.mp4" can be formed in the local storage path of the inference server 200.

[0029] Therefore, once the file to be processed is written, it is already located locally on the inference server 200 that performs the inference processing, and the inference server 200 does not need to read the file remotely from the independent shared storage or the front-end server 100.

[0030] S2: After the file to be processed is successfully written, the front-end server 100 generates a task message containing the local storage path and publishes the task message to the message middleware 300.

[0031] Front-end server 100 determines whether the file to be processed has been successfully written, based on the file output stream closing result, the file write interface return value, the file system refresh result, or the submission result returned by the network sharing protocol. After the file to be processed is successfully written, front-end server 100 generates a task message. The task message includes at least the local storage path of the file to be processed on inference server 200, enabling inference server 200 to directly locate the file to be processed after receiving the task message.

[0032] In some implementations, the task message may also include a task identifier, file type, upload time, service type, client identifier, callback address, priority information, or other information that facilitates subsequent processing.

[0033] After the front-end server 100 publishes the task message to the message middleware 300, it can return a response to the client indicating successful upload or task acceptance. At this point, the client upload process and the AI ​​model inference process of the inference server 200 are asynchronously decoupled, and the front-end server 100 can end the processing of the current upload request without waiting for the inference server 200 to complete the calculation.

[0034] Specifically, the file content itself is delivered to the local storage path of the inference server 200 via a network sharing protocol, while the message broker 300 only transmits task description information and file path information. Compared to directly transmitting large files in the message broker 300, this method avoids the message broker 300 carrying large file traffic. Compared to the inference server 200 subsequently reading the file from centralized storage, this method also ensures that the path in the task message directly points to a locally accessible location on the inference server 200, thereby reducing the data preparation process before the computation task starts.

[0035] S3: The inference server 200 obtains the task message from the message middleware 300 and reads the file to be processed through the local file system interface according to the local storage path in the task message.

[0036] The inference server 200 runs a task consumption process or analysis process, which acts as a consumer of the message middleware 300, continuously listening to its associated task queue. When there are tasks to be processed in the message middleware 300, the inference server 200 obtains the task message and parses the local storage path within it. Since this local storage path is located in the inference server 200's own file system, the inference server 200 can read the files to be processed through the local file system interface.

[0037] The local file system interface can be a file reading interface provided by the operating system or a file access interface provided by the runtime language.

[0038] S4: The inference server 200 performs AI model inference processing on the read files to be processed, and deletes the corresponding files to be processed in the local storage path after the processing is completed.

[0039] After reading the file to be processed, the inference server 200 inputs it into the AI ​​model's inference process. The inference server 200 can either pass the file path of the file to be processed to the AI ​​model's API call, or it can read the file content first, preprocess it, and then pass it to the model. After completing the inference, the AI ​​model generates the inference result. The inference result can be a classification label, confidence score, detection box, review conclusion, structured JSON result, or other data required by the business.

[0040] After inference processing is complete, the inference server 200 deletes the files corresponding to the task from the local storage path. This deletion can be performed directly by the analysis process executing the inference task after confirming the inference process is finished, or it can be performed by the analysis process calling the file management module. Since the files to be processed are located in the local storage path of the inference server 200, the deletion operation can also be completed through the local file system interface, without needing to access a separate shared storage system.

[0041] In this embodiment, the front-end server 100 receives files and writes them to the local storage path of the inference server 200. After successful file writing, a task message is published. The inference server 200 consumes the task message and reads the local file for AI model inference processing. After processing, the inference server 200 deletes the file to release local storage space. Since the file to be processed no longer passes through an independent centralized storage system, the data path is simplified from client to front-end server 100 to central storage to inference server 200 to client to front-end server 100 to inference server 200's local storage path. Due to this data path, the inference server 200 does not need to actively fetch files from external shared storage before executing inference tasks, thus reducing the CPU resources' burden of additional remote file access and network protocol processing load, thereby reducing the computational resource consumption of the inference server 200 by non-inference tasks.

[0042] In another embodiment, to accommodate scenarios with a large number of concurrent upload requests from clients, a front-end load balancing layer can be set up between the client and the front-end server 100. The front-end load balancing layer is used to receive file upload requests initiated by the client and distribute the upload requests to any one of the multiple front-end servers 100 according to a preset strategy.

[0043] The front-end load balancing layer can use hardware load balancing devices, cloud load balancing services, or software load balancing components.

[0044] The preset strategy can be round-robin, weighted round-robin, least connections, source address hash, business identifier hash, or other strategies that can achieve request distribution. Multiple front-end servers 100 together form an application server cluster. Each front-end server 100 is configured with a local mount path corresponding to the local storage path of the inference server 200, and a file upload service and a message middleware 300 client are deployed.

[0045] Since the front-end load balancing layer only distributes upload requests, and the files to be processed are ultimately written to the local storage path of the inference server 200, this implementation does not change the "write-to-be-in-place" data path in the above embodiment. In this way, the system can further improve the concurrent uploading capacity of client requests while maintaining local file reading by the inference server 200.

[0046] In another embodiment, the network sharing protocol can adopt the Network File System (NFS) protocol. The inference server 200 acts as an NFS server, exporting its local specified directory as a local storage path; the front-end server 100 acts as an NFS client, mounting the exported specified directory to its own local file system, generating a local mount path.

[0047] In one implementation, a specified directory, such as " / AI_FILE", is pre-created on the inference server 200. This directory is located on the inference server 200's local disk or local solid-state drive. The inference server 200 installs and starts an NFS server program, configuring the " / AI_FILE" directory as an exported directory that allows read and write access by the front-end server 100.

[0048] Front-end server 100 installs the NFS client component and mounts the " / AI_FILE" directory exported by inference server 200 to its own local directory, such as " / mnt / ai_share". After mounting, any write operations performed by front-end server 100 on the " / mnt / ai_share" directory will be written to the " / AI_FILE" directory of inference server 200 via the NFS protocol.

[0049] The NFS protocol provides the ability to mount remote file systems. For the file upload program on front-end server 100, " / mnt / ai_share" can be opened and written to as a local directory; for the analysis process on inference server 200, " / AI_FILE" is a directory in its local file system. This protocol layer mapping relationship allows the two servers to see different path entries, but they correspond to the same file entity in the local storage of inference server 200.

[0050] The NFS protocol implementation leverages the operating system's mature file system interface to perform cross-server writes. Both the upload program on the front-end server 100 and the analysis program on the inference server 200 can operate via ordinary file paths, reducing application-level modification costs. Furthermore, since the inference server 200 reads files from its own " / AI_FILE" directory during the inference phase, it does not need to fetch remote files through an NFS client; therefore, the data access path during the inference phase remains local.

[0051] In another embodiment, in order for the inference server 200 to accurately locate the file to be processed and for the message middleware 300 to deliver the task to the corresponding inference server 200 according to the specified route, the local storage path in the task message can be set to the local absolute path of the file to be processed on the inference server 200, and the task message can be bound to a dedicated task queue for the inference server 200 to listen to by specifying the switch and routing key.

[0052] In one implementation, after the front-end server 100 completes the file writing, the generated task message includes a file path field. This file path field stores the local absolute path on the inference server 200 side, such as " / AI_FILE / uuid1234.mp4". Although the actual path used by the front-end server 100 during writing may be " / mnt / ai_share / uuid1234.mp4", the task message does not use this path on the front-end server 100 side, but instead uses a path that the inference server 200 side can directly access. After parsing the task message, the inference server 200 can open the file using this local absolute path without performing path lookup or remote path conversion.

[0053] The message middleware 300 can employ a messaging system that supports exchanges, queues, and routing keys. The front-end server 100 publishes task messages to a designated exchange, such as a direct-connect exchange. This designated exchange delivers the task messages to the corresponding dedicated task queue based on the routing key. The analysis process of the inference server 200 continuously monitors this dedicated task queue, retrieving task messages that fall within its processing scope.

[0054] This implementation binds data paths and task paths together using a combination of local absolute paths and dedicated task queues. After a file to be processed is written to the local storage path of an inference server 200, the corresponding task message is routed to the queue listened to by the same inference server 200. Upon receiving the task, the inference server 200 directly reads the file based on the local absolute path, reducing the need for cross-node forwarding, path redirection, or secondary scheduling.

[0055] In another embodiment, the inference server 200 is equipped with a graphics processing unit (GPU). The inference server 200 transmits the data path of the file to be processed to the underlying AI model through a preset interface, and drives the GPU to perform inference calculations on the file to be processed to generate inference results.

[0056] The AI ​​model runtime environment and analysis service program can be deployed on the inference server 200. After receiving the task message from the message middleware 300, the analysis service program parses the local storage path and confirms the existence of the file to be processed through the local file system interface. Subsequently, the analysis service program can pass the local storage path as a parameter to the underlying AI model's call interface. The underlying AI model reads the file content according to the path, performs necessary decoding or preprocessing, and sends the preprocessed data to the GPU for inference computation.

[0057] In this implementation, the GPU of the inference server 200 primarily handles AI model inference computation, while the CPU is mainly used for task consumption, scheduling of small-scale file reads, data preprocessing, and GPU call management. Since the files to be processed are already located on the local storage path of the inference server 200, the inference server 200 does not need to perform remote file downloads or fetch data from centralized shared storage before starting inference, reducing the CPU usage due to network I / O and remote storage protocol processing. Therefore, the CPU of the inference server 200 can be used more for GPU task scheduling and model execution support, reducing the probability of GPU waiting due to file fetching.

[0058] In addition, after generating inference results, the inference server 200 can save or send the inference results to the result notification queue. After confirming that the inference results have been processed, the analysis process of the inference server 200 executes a file deletion command to clear the files to be processed in the local storage path, so as to free up the local storage space of the inference server 200.

[0059] In another embodiment, in order to reduce the risk of the inference service being affected by the local storage space of the inference server 200 being filled up, capacity detection and overload prevention processing can be performed before the front-end server 100 writes the file to be processed to the local storage path of the inference server 200.

[0060] Specifically, the front-end server 100 can detect the current remaining storage capacity of the storage volume where the local mount path is located. Since the local mount path is mapped to the local storage path of the inference server 200, the remaining storage capacity can reflect the available space status of the storage volume corresponding to the inference server 200.

[0061] The front-end server 100 can obtain the total capacity, used capacity and remaining capacity of the storage volume through the file system status query interface, or obtain capacity information through the status information provided by the operating system command, monitoring interface or network sharing protocol.

[0062] Front-end server 100 compares the current remaining storage capacity with a preset security threshold. The security threshold can be configured based on the local storage volume capacity of inference server 200, the average size of a single file to be processed, inference time, concurrent upload volume, and business fault tolerance requirements. For example, in a scenario with a local storage volume capacity of 1TB, the security threshold can be set to 100GB; it can also be set proportionally, such as using a remaining capacity falling below 10% of the total capacity as a trigger condition. The above values ​​are merely illustrative examples; in actual deployment, they can be adjusted according to file size distribution and inference processing speed.

[0063] If the remaining storage capacity is not lower than the preset security threshold, the front-end server 100 continues to perform file writing operations, writing the file to be processed to the local mount path. If the remaining storage capacity is lower than the preset security threshold, the front-end server 100 stops writing the file to be processed to the inference server 200 and triggers the traffic scheduling policy.

[0064] Traffic scheduling strategies can include two processing methods. The first method involves routing the file to be processed to the local storage path of another standby inference server 200 that is operating at normal capacity. In this method, the other standby inference server 200 also maps its local storage path to the local mount path of the front-end server 100. When the front-end server 100 detects that the current inference server 200's capacity is insufficient, it selects a standby inference server 200 with sufficient remaining capacity, writes the file to be processed to the local mount path corresponding to that standby inference server 200, and generates a task message containing the local storage path of that standby inference server 200. The task message is then published to the task queue corresponding to that standby inference server 200, allowing the standby inference server 200 to consume the task.

[0065] The second approach is to return a rate-limiting retry response to the client. When there is no available backup inference server 200, or when business policies do not allow task transfer, the front-end server 100 can return a response to the client indicating that the current system is busy or storage capacity is insufficient, prompting the client to retry later. The rate-limiting retry response may include the retry time, error code, or task not accepted status, allowing the client or upstream system to control retry operations.

[0066] This capacity detection and overload prevention mechanism assesses risks before files are written to the inference server 200, preventing continuous writing of files from exhausting the server's local storage. Since the inference server 200 also uses local storage for data reading and file cleanup for AI inference tasks, a full storage volume could lead to new file write failures, result file saving failures, or system service anomalies. By performing capacity detection and traffic scheduling on the front-end server 100, overload pressure can be blocked before writing, ensuring the inference server 200 maintains available local storage.

[0067] In summary, this invention discloses a computational task processing method for AI inference services. The deployment and operation of the above method will be comprehensively explained below using an online AI content review service as an example.

[0068] The system consists of one inference server 200 and two front-end servers 100. Inference server 200 deploys an AI content moderation model and is equipped with a GPU for performing image or video content moderation inference. The two front-end servers 100 receive image or video files uploaded by clients. A message middleware 300 is also deployed to transmit analysis task messages.

[0069] The internal network address of inference server 200 can be set to 172.16.12.32, and its local storage path is " / AI_FILE".

[0070] The internal network addresses of the two front-end servers 100 can be 172.16.12.36 and 172.16.12.37, respectively. The inference server 200 installs the network file system server component, creates the " / AI_FILE" directory, and configures this directory as a shared directory that allows read and write access from both front-end servers 100.

[0071] Front-end server 100 installs the network file system client component and mounts the " / AI_FILE" directory exported by inference server 200 to its own " / mnt / ai_share" directory. Therefore, files written to the " / mnt / ai_share" directory by front-end server 100 are actually stored in the " / AI_FILE" directory of inference server 200.

[0072] An analysis service program and an AI model library are deployed on the inference server 200. The analysis service program connects to the message middleware 300 and listens to a dedicated task queue. The AI ​​model library can be invoked by the analysis service program through a preset interface. For example, the analysis service program can pass the local absolute path of the file to be processed on the inference server 200 to the underlying model library, which will then perform video decoding, image frame extraction, model inference, and result generation.

[0073] The front-end access point can expose a unified domain name through a load balancing layer. The client initiates a file upload request to this domain, carrying the video file to be reviewed. The load balancing layer forwards this request to one of the front-end servers 100 according to a round-robin strategy. Front-end server 100 receives the video file stream and generates a unique filename, such as "uuid1234.mp4". Subsequently, front-end server 100 writes the file stream to its local mount path " / mnt / ai_share / uuid1234.mp4". Since this path is the mount path of inference server 200 " / AI_FILE", the file is ultimately created on inference server 200 as " / AI_FILE / uuid1234.mp4".

[0074] After the file is successfully written, the front-end server 100 constructs a task message. The task message can include a task identifier, a file path, and a timestamp. For example, the task identifier is "task_001" and the file path is " / AI_FILE / uuid1234.mp4". Here, the file path uses the local absolute path on the side of the inference server 200, rather than the mounted path on the side of the front-end server 100. The front-end server 100 publishes the task message to a specified switch of the message middleware 300 and uses a routing key corresponding to the inference server 200, so that the task message is delivered to a dedicated task queue listened by the inference server 200. The front-end server 100 can then return a task accepted response to the client, and the client can query the subsequent results through the task identifier.

[0075] The analysis service program on the inference server 200 obtains the task message from the dedicated task queue and resolves the local storage path " / AI_FILE / uuid1234.mp4". The analysis service program reads the file under this path through the local file system interface. Since the file is located in the local file system of the inference server 200, the inference server 200 does not need to download the file from the remote centralized storage during the reading process, nor does it need to access the front-end server 100. The analysis service program passes the file path into the AI model call interface, and the underlying model uses the GPU to perform content review inference and generates an inference result including a review conclusion, a confidence level, and a task identifier.

[0076] After the inference result is generated, the analysis service program saves the inference result to the database or sends the inference result to the result notification queue. After confirming that the inference result has been saved or sent, the analysis service program deletes the file " / AI_FILE / uuid1234.mp4" and releases the local storage space of the inference server 200. The client can poll the query interface according to the task identifier or obtain the final review result through other notification mechanisms.

[0077] It can be seen that the client upload traffic is borne by the front-end server 100, the AI model inference is borne by the inference server 200, and the task coordination is borne by the message middleware 300. The file to be processed is directly written to the local storage path of the inference server 200 after being received by the front-end server 100, and the task message only carries lightweight information such as the path. After consuming the task, the inference server 200 reads the file according to the local path and performs inference. After the inference is completed, the source file is cleaned up. This processing flow reduces the centralized shared storage link and also avoids the inference server 200 actively pulling the file before calculation, so that file access, task scheduling, AI inference, and file cleaning form a continuous processing closed loop.

[0078] In another preferred embodiment of the present invention, a computational task processing system for AI inference services is also disclosed, which includes a front-end server and an inference server, wherein the front-end server and the inference server process user-uploaded files based on the computational task processing method in the above embodiment.

[0079] The present invention also discloses another computing task processing system, which includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for performing the computing task processing method as described above. The processor may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, used to execute related programs to implement the functions required by the modules in the computing task processing system of the embodiments of this application, or to execute the computing task processing method of the method embodiments of this application.

[0080] This invention also discloses a computer-readable storage medium comprising a computer program that can be executed by a processor to perform the computational task processing method described above. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be read-only memory (ROM), random access memory (RAM), or magnetic media, such as floppy disks, hard disks, magnetic tapes, magnetic disks, or optical media, such as digital versatile discs (DVDs), or semiconductor media, such as solid-state disks (SSDs).

[0081] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned computational task processing method.

[0082] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A computational task processing method for AI inference services, characterized in that, include: The front-end server receives the file to be processed uploaded by the client and writes the file to be processed into the local storage path of the inference server through a network sharing protocol; wherein, the local storage path of the inference server is pre-mapped to the local mount path of the front-end server. After the file to be processed is successfully written, the front-end server generates a task message containing the local storage path and publishes the task message to the message middleware. The inference server obtains the task message from the message middleware and reads the file to be processed through the local file system interface according to the local storage path in the task message; The inference server performs AI model inference processing on the read file to be processed, and deletes the corresponding file to be processed in the local storage path after the processing is completed; The method for the front-end server to receive files to be processed uploaded by the client includes: The front-end server receives file upload requests initiated by the client through the front-end load balancing layer; The front-end load balancing layer distributes the upload request to any one of the application server clusters with multiple front-end servers according to a preset strategy. The local storage path in the task message is the local absolute path of the file to be processed on the inference server. The front-end server publishes the task message to a designated switch of the message middleware and binds it to a dedicated task queue for the inference server to listen to via a routing key.

2. The computational task processing method according to claim 1, characterized in that, The network sharing protocol is the Network File System (NFS) protocol. The inference server acts as an NFS server, exporting its local specified directory to the local storage path; The front-end server, acting as an NFS client, mounts the exported specified directory to its own local file system, generating the local mount path.

3. The computational task processing method according to claim 1, characterized in that, The inference server is equipped with a graphics processing unit (GPU). The inference server transmits the data path of the file to be processed to the underlying AI model through a preset interface, and drives the GPU to perform inference calculations on the file to be processed to generate inference results.

4. The computational task processing method according to claim 1, characterized in that, After generating the inference result, the inference server saves or sends the inference result to the result notification queue. After confirming that the inference result processing is complete, the analysis process of the inference server executes a file deletion command to clear the files to be processed under the local storage path, thereby freeing up the local storage space of the inference server.

5. The computational task processing method according to claim 1, characterized in that, Before the front-end server writes the file to be processed to the local storage path of the inference server via a network sharing protocol, it also includes a capacity detection and overload prevention method: The front-end server detects the current remaining storage capacity of the storage volume where the local mount path is located; If the current remaining storage capacity is lower than a preset security threshold, the front-end server will suspend writing the file to be processed to the inference server and trigger a traffic scheduling policy to route the file to be processed to the local storage path of another standby inference server in normal capacity, or return a rate-limiting retry response to the client.

6. A computational task processing system for AI inference services, characterized in that, It includes a front-end server and an inference server, wherein the front-end server and the inference server process the user-uploaded files based on the computing task processing method according to any one of claims 1 to 5.

7. A computational task processing system for AI inference services, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for performing the computational task processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Includes a computer program that can be executed by a processor to perform the computational task processing method as described in any one of claims 1 to 5.

Images