Multi-process task reasoning system and method and computer equipment

Abstract

The invention discloses a multi-process task reasoning system and method and computer equipment. The multi-process task reasoning system comprises a scheduling process and a plurality of computing power nodes; the scheduling process responds to an external task reasoning request, analyzes the task and queries the current working state of each computing power node; the scheduling process selects the computing power node for executing task reasoning based on the analysis result of the task and the current working state of each computing power node; and the computing power node executes reasoning of the task and sends a reasoning result of the task to the scheduling process after execution is completed so as to update the processing state of the task. In this way, the flow sharing efficiency, the resource utilization rate and the stability caused by task reasoning single-process design can be improved.

Description

Technical Field

[0001] This application relates to the field of visual algorithms, and in particular to a multi-process task reasoning system, method and computer device. Background Technology

[0002] With the application of artificial intelligence technology across various industries, the demand for business applications such as visual algorithms has surged, creating an urgent need for efficient, flexible, and stable inference engines to support algorithm operation. However, existing inference engines generally adopt a single-process integrated design, placing the entire process logic, including video streaming, decoding, inference, and alarming, within the same working process. This leads to the following core technical challenges: Existing technologies only support intra-process video stream sharing and cannot achieve stream reuse across processes or GPUs. Multiple inference tasks from the same video source need to repeatedly stream and decode, resulting in redundant consumption of network bandwidth and computing resources. If a single-process-bound-single-GPU model is adopted, streaming and inference must use the same GPU, making it impossible to flexibly allocate hardware resources according to task characteristics, resulting in underutilization of the GPU's decoding / inference capabilities. If a single-process-bound-multiple-GPU model is adopted, the process needs to handle multiple streams and multiple algorithm tasks simultaneously. If one video stream fails or an algorithm plugin crashes, the entire process will exit, causing widespread task interruption and weak system fault tolerance. These problems restrict the application of inference engines in large-scale, high-concurrency scenarios, and urgently require architectural innovation to resolve the contradiction between resource reuse, hardware scheduling, and stability. Summary of the Invention

[0003] This application provides a multi-process task inference system, method, and computer device to solve the core problems of low flow sharing efficiency, insufficient resource utilization, and poor stability caused by the single-process design of task inference.

[0004] To address the aforementioned technical problems, this application provides a multi-process task inference system, comprising a scheduling process and multiple computing nodes. The scheduling process responds to an external task inference request by parsing the task and querying the current working status of each computing node. Based on the parsing result of the task and the current working status of each computing node, the scheduling process selects a computing node to execute the task inference. The computing node executes the task inference and, upon completion, sends the inference result of the task to the scheduling process to update the task's processing status.

[0005] In some embodiments, each computing node includes a daemon process and multiple processing units; the daemon process is used to receive query requests sent by the scheduling process, obtain the working status of each processing unit to obtain the current working status of the computing node to which it belongs, and automatically generate a new processing unit when the processing unit exits the running state.

[0006] In some embodiments, the processing unit includes a streaming process and an inference process; the streaming process pulls streaming data from an external video source according to the instructions contained in the task, and converts the streaming data into raw video frames; the inference process performs inference for the task through an algorithm plugin.

[0007] In some embodiments, the computing node further includes a node shared memory region, through which the processing units communicate with each other.

[0008] In some embodiments, the processing unit further includes a video memory shared region, through which the fetching process and the inference process share the original video frame pixel data.

[0009] In some embodiments, the node shared memory region includes a frame queue consisting of a circular buffer, the fetching process puts the pixel data into the frame queue, and the inference process obtains the pixel data from the frame queue through real-time consumption.

[0010] In some embodiments, the stream fetching process, the inference process, and the daemon process communicate directly via sockets, and the daemon process and the scheduling process communicate directly via sockets.

[0011] In some embodiments, the inference process includes multiple types of algorithm plugins, and the inference process calls the corresponding algorithm plugins to perform the inference of the task based on the requirements of the task.

[0012] In some embodiments, the multi-process task inference system is deployed with at least two scheduling processes, and each scheduling process forms a master-slave relationship; when the master scheduling process exits abnormally, the backup scheduling process automatically takes over the task processing progress of the master scheduling process and becomes the new master scheduling process.

[0013] In some embodiments, the node registration information and task status data of the computing nodes are stored in a third-party key-value storage service.

[0014] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a multi-process task inference method, which is used in the multi-process task inference system described above, comprising: responding to an external task inference request, querying the current working status of each computing power node; selecting a computing power node to infer the task based on the query result and the inference requirements of the task; and after the inference of the task is completed, feeding back the inference result of the task to the outside.

[0015] This application also provides a computer device on which the multi-process task inference system as described above is deployed.

[0016] The beneficial effects of this application are as follows: Unlike existing technologies, this application discloses a multi-process task inference system, method, and computer device. By scheduling a process to respond to external task inference requests, it parses the task and queries the current working status of each computing node; it achieves centralized task management, and the scheduling process can uniformly record the streaming status of video sources. Inference tasks for the same video source can reuse existing streaming processes, directly solving the problem of low efficiency in video stream sharing in existing technologies, saving network transmission and video decoding resources; it obtains real-time data such as GPU utilization, memory usage, and working process status of computing nodes, providing data support for subsequent node selection, breaking the limitations of insufficient hardware resource utilization in existing technologies, and laying the foundation for on-demand resource allocation. The scheduling process selects the computing node to execute the task inference based on the task parsing results and the current working status of each computing node; it supports the separate configuration of streaming and inference GPUs, completely solving the limitation of binding streaming and inference GPUs in existing technologies, and significantly improving hardware flexibility and utilization. The computing nodes perform task inference and send the inference results to the scheduling process after execution to update the task's processing status; centralized scheduling reuses the streaming process of the same video source to avoid repeated streaming decoding; collaboratively realizes the efficiency, flexibility and stability of the inference engine, which is significantly better than the single-process integrated design of existing technologies. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein: Figure 1 This is a schematic diagram of the structure of an embodiment of a multi-process task inference system provided in this application; Figure 2 Is it like this? Figure 1 A schematic diagram of the structure of a computing node in the system shown in the figure; Figure 3 Is it like this? Figure 2 A schematic diagram of the structure of one embodiment of the processing unit in the system shown; Figure 4 This is a flowchart illustrating an embodiment of the multi-process task reasoning method provided in this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," and "third" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] See Figure 1 , Figure 1 This is a schematic diagram of an embodiment of a multi-process task inference system 100 provided in this application. The multi-process task inference system 100 includes a scheduling process 110 and multiple computing nodes 120.

[0022] The scheduling process 110 can deploy multiple instances to form a master-slave relationship. It is responsible for processing task inference requests, managing and scheduling tasks, responding to external task inference requests, parsing tasks, and querying the current working status of each computing node 120. Based on the task parsing results and the current working status of each computing node 120, the scheduling process 110 selects the computing node 120 to execute the task inference.

[0023] Computing node 120 performs task inference and sends the inference result to scheduling process 110 after completion to update the task's processing status. Each computing node 120 deploys a daemon process 121 and several worker processes; daemon process 121 is responsible for resource management and keeping worker processes alive; worker processes are bound to a computing card, streaming process 1221 performs video streaming and decoding, and inference process 1222 calls algorithm plugins to complete inference.

[0024] The multi-process task inference system 100 includes at least two scheduling processes 110, which are responsible for task reception, parsing and allocation of computing power nodes 120, and have redundancy capabilities; the multi-process task inference system 100 also includes multiple computing power nodes 120, which serve as computing power carriers for task inference and are responsible for executing specific computing tasks.

[0025] The redundant design of at least two scheduling processes 110 and the configuration of multiple computing nodes 120 avoid the risk of single point of failure and ensure the continuity of task processing.

[0026] The scheduling process 110 responds to external task inference requests, decomposes task attributes such as computing power requirements, time constraints, and task type, and generates parsing results; it queries the current working status of all computing power nodes 120 in real time, such as idle / busy, remaining computing power, and task load; based on the matching degree between the task parsing results and the node status, such as the adaptability between the task computing power requirements and the node's remaining computing power, it selects the optimal computing power node 120 to execute the task; it receives the inference results returned by the computing power node 120 and synchronously updates the task processing status, such as completed, failed, or pending retry.

[0027] Multiple scheduling processes (110 in total) receive tasks in parallel, avoiding bottlenecks caused by single scheduling. A dynamic node allocation strategy fully utilizes computing resources and reduces task queuing time. A precise matching strategy between task requirements and node status avoids computing power mismatch and improves resource utilization efficiency.

[0028] Computing node 120 receives the task assigned by scheduling process 110, starts inference computation, and after completing the computation, sends the inference result to scheduling process 110 and updates its own status, such as changing from busy to idle. Real-time status query combined with the task status update mechanism enables rapid response to fluctuations in task volume, ensuring stable system operation.

[0029] Specifically, the multi-process task inference system 100 deploys at least two scheduling processes 110, and the scheduling processes 110 form a master-slave relationship. When the master scheduling process 110 exits abnormally, the slave scheduling process 110 automatically takes over the task processing progress of the master scheduling process 110 and becomes the new master scheduling process 110.

[0030] The multi-process task inference system 100 is deployed with multiple scheduling processes 110, and the scheduling processes 110 form a master-slave relationship: one of them is the running master scheduling process 110, and the rest are standby scheduling processes 110 in a standby state.

[0031] Among them, the main scheduling process 110 is responsible for daily task processing, including task reception, parsing, allocation of computing power nodes 120 and progress management, while at least one backup scheduling process 110 is in a real-time monitoring state and does not participate in daily task processing.

[0032] The primary scheduling process 110 continuously pushes core information such as the list of unfinished tasks, node allocation records, and the status of computing nodes 120 to the backup scheduling process 110 through the status synchronization module, ensuring that the status of the backup process is completely consistent with that of the primary process. The backup scheduling process 110 determines in real time whether the primary process is running normally through heartbeat detection or process liveness monitoring. When the primary scheduling process 110 exits abnormally, the backup scheduling process 110 triggers a switching mechanism, restores the task processing context based on the synchronized status information, takes over all unfinished tasks and node management responsibilities of the primary process, and becomes the new primary scheduling process 110. If there are multiple backup scheduling processes 110, the remaining backup processes are automatically adjusted to the new backup scheduling role to maintain the redundancy of the primary-backup architecture.

[0033] The primary-backup redundancy design avoids single points of failure. When the primary process fails, the backup process quickly takes over to ensure continuous system operation. The real-time status synchronization mechanism ensures that task progress is not lost when the backup process takes over, and unfinished tasks can be seamlessly continued. Automatic switching requires no manual intervention, reducing fault response time and manual maintenance costs. The backup process only listens and does not participate in daily tasks, so it does not affect the processing efficiency of the primary process, while maintaining the ability to take over at any time.

[0034] Optionally, the node registration information and task status data of computing node 120 are stored in a third-party key-value storage service.

[0035] Computing node 120 serves as the execution carrier for task inference, responsible for writing its own registration information to third-party key-value services and updating the task execution status in real time.

[0036] Third-party key-value storage services serve as a unified data persistence layer, storing node registration information such as node ID, computing power specifications, network address, and liveness status, as well as task status data such as task ID, associated node ID, execution progress, and result status. They feature high availability and low latency access characteristics.

[0037] As the decision-making body for task allocation, scheduling process 110 obtains node information and task status by querying third-party key-value services, thereby realizing node selection and task management.

[0038] After computing node 120 starts, it generates registration information containing its own attributes and writes it to a third-party key-value service with the node ID as the key and the attribute set as the value. After receiving a task request, scheduling process 110 reads the registration information and current task status of all nodes from the key-value service, and selects idle nodes with matching computing power to assign tasks. During the execution of tasks, computing node 120 updates the task status data in the key-value service with the task ID as the key after each stage is completed, such as initialization, computation in progress, and completion. If computing node 120 exits abnormally, scheduling process 110 identifies abnormal nodes by periodically querying the node liveness status in the key-value service, and reassigns unfinished tasks to other available nodes based on the latest task status in the key-value service.

[0039] The third-party key-value service serves as a single data source, allowing all components to share the same set of node and task data, thus avoiding data redundancy and inconsistency issues caused by local storage. Adding new computing nodes 120 or scheduling processes 110 does not require modifying the local storage logic; integration can be completed directly by connecting to the third-party key-value service, reducing system expansion costs.

[0040] When computing node 120 malfunctions, scheduling process 110 can quickly obtain the latest task status from the key-value service without re-initializing the context, thus shortening the task recovery time. The storage layer is separated from the business layer, and computing node 120 and scheduling process 110 only interact through the key-value service API, without needing to pay attention to data storage details, thereby improving system maintainability.

[0041] Scheduling process 110 selects the optimal node based on the real-time node status in the key-value service, avoiding wasted computing power and improving task processing efficiency. Specifically, see Figure 2 Each computing node 120 includes a daemon process 121 and multiple processing units 122.

[0042] The daemon process 121 is used to receive query requests sent by the scheduling process 110, obtain the working status of each processing unit 122, so as to obtain the current working status of the computing power node 120 to which it belongs, and automatically generate a new processing unit 122 when the processing unit 122 exits the running state.

[0043] As the management core of computing node 120, daemon process 121 is responsible for interacting with scheduling process 110, such as receiving query requests, returning node status, monitoring the life cycle of processing unit 122 and automatic recovery processing unit 122.

[0044] The processing unit 122 contains multiple processing units 122, which serve as the execution carrier for task reasoning, are responsible for executing the specific tasks assigned by the scheduling process 110, and have parallel processing capabilities.

[0045] The scheduling process 110 sends a query request to the daemon process 121 of the computing node 120. The daemon process 121 traverses all processing units 122, collects the working status of each unit, and summarizes the current working status of the node and returns it to the scheduling process 110. After receiving the task assigned by the scheduling process 110, the daemon process 121 distributes the task to the idle processing unit 122. The processing unit 122 executes the task and synchronizes the execution progress with the daemon process 121. If the processing unit 122 exits the running state, the daemon process 121 identifies the abnormality through heartbeat detection or process status monitoring, and automatically calls a new processing unit 122 to replace the abnormal unit, so as to maintain the continuity of the node's computing power.

[0046] The daemon process 121 manages the status of the processing unit 122 in a unified manner, and the scheduling process 110 does not need to interact directly with each processing unit 122, reducing communication overhead and query latency; the daemon process 121 automatically detects and restarts the processing unit 122 in case of abnormality, avoiding the loss of node computing power or task interruption caused by the exit of a single processing unit 122.

[0047] The multiprocessing unit 122 executes tasks in parallel, and the daemon process 121 dynamically allocates tasks to idle units, making full use of node hardware resources. The daemon process 121 shields the details of the processing unit 122, and the scheduling process 110 only needs to interact with the daemon process 121, reducing system coupling.

[0048] Further, see Figure 3 The processing unit 122 includes a stream fetching process 1221 and an inference process 1222.

[0049] The streaming process 1221 retrieves streaming data from an external video source according to the instructions contained in the task and converts the streaming data into raw video frames. The inference process 1222 performs task inference through the algorithm plugin.

[0050] The streaming process 1221 serves as the data input layer. It pulls streaming data from external video sources according to the instructions contained in the task, converts it into raw video frames, and provides standardized input for the inference process 1222.

[0051] Inference process 1222, as the task execution layer, loads algorithm plugins, such as object detection and behavior recognition plugins, to perform inference calculations on the original video frames and output the task results.

[0052] The streaming process 1221 parses the video source information in the task instruction and pulls the encoded stream data from external video sources, such as cameras, cloud storage, and live streams. The streaming process 1221 decodes the encoded stream data and converts it into raw video frames in a unified format. The streaming process 1221 passes the raw video frames to the inference process 1222. The inference process 1222 loads the specified algorithm plugin, performs inference calculations on the frame data, generates task results, and provides feedback.

[0053] The fetching process 1221 focuses on data fetching and format conversion, while the inference process 1222 focuses on algorithm calculation. The two have a clear division of labor, and upgrades to the inference algorithm do not affect the fetching process. Adjustments to the fetching logic do not depend on the inference plugin, thus improving system maintainability. The fetching process 1221 fetches and converts frame data in parallel, while the inference process 1222 performs algorithm calculations synchronously. This avoids performance bottlenecks caused by a single process handling I / O and calculations simultaneously, thereby improving overall throughput.

[0054] The inference process 1222 adopts a plug-in design. When adding or replacing algorithms, only the plug-in needs to be updated. There is no need to modify the streaming process 1221 or other modules of the processing unit 122, which reduces the algorithm iteration cost. The streaming process 1221 uniformly converts the original video frame format. The inference process 1222 does not need to deal with format compatibility issues, ensuring the standardization of input data and reducing the inference error rate.

[0055] Optionally, the computing node 120 also includes a node shared memory area, through which the processing units 122 communicate with each other.

[0056] The shared area within the node serves as a data interaction layer between processing units 122, storing core data that needs to be shared, such as original video frames and process status identifiers, and providing a low-latency access interface using memory mapping.

[0057] The processing unit 122 that needs to send data writes the shared data to a specified address in the shared memory region and sets the status flag to "readable". The shared memory region ensures that the data is visible in real time through a memory mapping mechanism, without the need for additional data copying. The processing unit 122 that receives data polls or listens to the status flag. When the flag is "readable", it reads the data from the specified address and updates the status flag to "read".

[0058] The shared memory region uses direct memory access, avoiding kernel / user mode switching and data copying for inter-process communication, thus reducing process communication latency. The memory mapping mechanism supports high-bandwidth transmission and is suitable for high-frequency interaction scenarios with large amounts of data, such as raw video frames, thereby improving data interaction throughput and effectively enhancing task processing efficiency.

[0059] The processing unit 122 does not need to implement complex network communication protocols; it only needs to read and write data through memory addresses, which reduces code complexity and maintenance costs and simplifies process communication logic.

[0060] Shared memory regions use status flags to synchronize data, avoiding data conflicts caused by multiple processes reading and writing simultaneously.

[0061] Processing unit 122 interacts indirectly through shared memory, without needing to directly hold the handle of the other process, thereby improving the modularity and scalability of the system and decoupling process dependencies.

[0062] Optionally, the processing unit also includes a video memory shared area, through which the fetching process 1221 and the inference process 1222 share the original video frame pixel data.

[0063] The shared video memory region serves as a dedicated shared layer for the original video frame pixel data within the processing unit 122. It leverages GPU memory to achieve low-latency, high-bandwidth data storage and access, avoiding data copying across CPU-GPU devices.

[0064] The fetching process 1221 pulls streaming data from an external video source, decodes it to obtain the original video frame pixel data, and writes the data directly to a specified address in the shared memory region through the GPU memory mapping interface. The shared memory region ensures that the pixel data is visible to the inference process 1222 in real time through the GPU hardware's memory consistency mechanism, without the need for CPU memory transfer. After loading the algorithm plugin, the inference process 1222 directly reads the original video frame pixel data from the shared memory region and performs inference calculations.

[0065] The original video frame pixel data is directly stored in the shared area of ​​video memory, avoiding redundant data copying between CPU memory and GPU memory, reducing memory bandwidth usage, and reducing cross-device data copying overhead.

[0066] The original video frame pixel data is stored only in the shared area of ​​the video memory, without occupying the CPU memory at the same time, thus optimizing the utilization of system resources. The shared area of ​​the video memory utilizes the high bandwidth characteristics of the GPU video memory to support high-frequency interaction of large-volume video frame data, meeting the high throughput requirements of real-time video inference.

[0067] Inference process 1222 directly accesses video memory data, with data access latency far lower than CPU-GPU cross-device transmission, improving the real-time response speed of video inference and enhancing inference real-time performance.

[0068] Specifically, the node shared memory region includes a frame queue consisting of a circular buffer. The fetching process 1221 puts pixel data into the frame queue, and the inference process 1222 retrieves pixel data from the frame queue through real-time consumption.

[0069] The circular buffer frame queue, as a dedicated layer for pixel data transfer, is located within the shared memory area of ​​the nodes. It adopts a fixed-size circular structure with the beginning and end logically connected, and achieves asynchronous data interaction between the fetching and inference processes 1222 through the separation of read and write pointers.

[0070] The fetching process 1221 writes the decoded pixel data to the tail pointer position of the frame queue. The write pointer moves forward, and if it reaches the end of the queue, it jumps to the head of the queue, realizing space circular reuse. The frame queue uses fixed memory space and does not require dynamic memory allocation. When the queue is full, the fetching process 1221 pauses writing or overwrites old data according to a strategy. The inference process 1222 reads pixel data from the head pointer position of the frame queue, and the read pointer moves forward. When the queue is empty, the inference process 1222 blocks and waits for new data. The read and write pointer states are maintained through atomic operations to ensure that there is no data contention during concurrent read and write operations by multiple processes.

[0071] The circular structure reuses fixed memory in a circular manner, avoiding frequent dynamic memory allocation / release and eliminating memory fragmentation; the separation of read and write pointers enables asynchronous parallel interaction, allowing the streaming and inference processes 1222 to operate simultaneously without redundant data copying; the inference process 1222 consumes frame queue data in real time, adapting to the low-latency processing requirements of continuous data such as video streams; fixed space limits data accumulation and avoids memory overflow; atomic operations ensure concurrent read and write consistency, eliminating the risk of data loss / error and enhancing system stability.

[0072] Optionally, the stream fetching process 1221, the inference process 1222, and the daemon process 121 communicate directly through sockets, and the daemon process 121 communicates directly with the scheduling process 110 through sockets.

[0073] The socket communication layer uses the TCP / UDP protocol to implement direct inter-process communication, supports cross-node network transmission, and provides a unified communication interface for all components.

[0074] The scheduling process 110 sends task instructions to the daemon process 121 via a socket. After receiving the instructions, the daemon process 121 starts or wakes up the stream retrieval process 1221 and the inference process 1222, and reports the node ready status to the scheduling process 110 through a third-party key-value service.

[0075] The daemon process 121 sends control commands to the stream fetching / inference process 1222 via a socket. The stream fetching / inference process 1222 reports its real-time status to the daemon process 121 via a socket. If the process is abnormal, the daemon process 121 sends a restart command via a socket to restore the operation of the processing unit 122.

[0076] After the streaming process 1221 completes video decoding, it sends a task start command to the inference process 1222 via a socket. After the inference process 1222 completes the inference task, it sends the inference result back to the streaming process 1221 via a socket. The streaming process 1221 summarizes the results and reports them to the daemon process 121 via a socket.

[0077] After integrating the task results of the processing unit 122 with the node status, the daemon process 121 reports the results to the scheduling process 110 via a socket; after receiving the results, the scheduling process 110 issues subsequent task instructions.

[0078] The socket is based on the TCP / UDP network protocol. The scheduling process 110 can be deployed on remote nodes to realize the distributed management of the computing nodes 120. The processes communicate with each other through network ports, without shared memory dependencies, which facilitates the independent upgrading and replacement of individual components. The TCP protocol is used to ensure the reliability of data transmission, or the UDP protocol to meet the low latency requirements. All processes use socket communication, which simplifies development and maintenance costs and avoids compatibility issues of multiple communication methods. The daemon process 121 communicates directly with the scheduling process 110, supporting the dynamic issuance, pausing and resumption of tasks, and adapting to the task adjustment needs in complex scenarios.

[0079] Specifically, the inference process 1222 includes multiple types of algorithm plugins. Based on the requirements of the task, the inference process 1222 calls the corresponding algorithm plugins to perform the inference of the task.

[0080] The various algorithm plugins are independently encapsulated algorithm function modules. Each type of plugin corresponds to a specific inference task and interacts with the inference process through a unified interface layer.

[0081] The unified interface layer defines the input and output formats, initialization / execution / destruction methods of plugins, providing standardized interaction specifications for the inference process 1222 and plugins.

[0082] The inference process 1222 receives task instructions from the stream fetching process 1221 or the daemon process 121, and parses out the task type, input data format and parameters; the inference process 1222 matches the corresponding algorithm plugin according to the task type, loads the plugin through a unified interface and completes initialization; the inference process 1222 passes the input data to the plugin, and the plugin performs inference calculations; the plugin returns the inference result, and the inference process 1222 summarizes the results and feeds them back to the external component; if the task ends, the inference process 1222 unloads the plugin and releases resources.

[0083] The plug-in design allows the inference process to dynamically call the corresponding plug-in according to the task requirements, so that different types of inference tasks can be handled without modifying the core code; the algorithm plug-in is independently encapsulated, and the development, testing and upgrading process does not affect the core logic of the inference process, and troubleshooting only needs to focus on the specific plug-in.

[0084] New algorithms only require the development of plugins that conform to a unified interface, without the need to refactor the inference process 1222, quickly supporting emerging business needs; the inference process 1222 loads plugins on demand, and unused plugins do not occupy memory / GPU resources, adapting to resource-constrained computing nodes 120 scenarios; the plugin architecture isolates the iteration cycles of different algorithms, old plugins can continue to be used, and new plugins can be developed in parallel, avoiding version conflicts.

[0085] See Figure 4 This application also provides a multi-process task inference method, which is used in the multi-process task inference system 100 as described above, and includes the following steps: 01: In response to an external task inference request, query the current working status of each computing node 120.

[0086] After the system responds to the task inference request initiated by the outside, it immediately queries the current working status of each computing node 120, such as load rate, resource idleness, and algorithm plugin support status.

[0087] 02: Based on the query results and the reasoning requirements of the task, select computing power node 120 for the reasoning task.

[0088] Based on the node status query results and combined with the inference requirements of the task, such as the required computing power scale and algorithm type adaptability, the optimal computing power node 120 is selected and determined to complete the allocation of the inference task.

[0089] 03: After the reasoning for the task is completed, the reasoning results of the task are fed back to the outside.

[0090] Once computing node 120 completes the inference task, the system will promptly send the inference results back to the external requester, thus ending the entire processing flow.

[0091] By querying the working status of nodes, tasks are avoided from being assigned to overloaded nodes, thus achieving balanced scheduling of computing resources; matching suitable nodes based on task requirements reduces task waiting time and resource waste, and speeds up the completion of inference tasks; forming a complete request-response closed loop ensures that external requests receive timely and accurate results, improving service quality and user experience.

[0092] This application also provides a computer device on which the multi-process task inference system 100 as described above is deployed.

[0093] The computer device deploys a multi-process task inference system 100, whose core components include: a daemon process 121 responsible for managing the lifecycle of processes such as fetching and inference, and monitoring the device status; a set of processing units 122 containing fetching process 1221 and inference process 1222; a scheduling process 110 that responds to external task requests, queries node status and allocates optimal inference resources; and a communication layer that enables direct inter-process communication through sockets. The communication layer is used to ensure efficient transmission of instructions and data.

[0094] This computer equipment is suitable for AI inference tasks in various scenarios such as video analysis, image recognition, and natural language processing, providing efficient and stable computing power support for enterprise-level AI applications.

[0095] Unlike existing technologies, this application's multi-process architecture layered design splits the inference engine into three categories: scheduling process 110, daemon process 121, and worker processes. This decouples the streaming and inference processes, requiring only one streaming path for the same video source, saving network transmission and decoding resources. Simultaneously, it establishes process isolation, limiting the impact of unstable algorithm plugin failures and preventing global task interruption due to single-process crashes. The scheduling process 110 employs a master-slave mechanism, deploying multiple instances of scheduling process 110 to form a master-slave relationship. When the master process fails, the backup process automatically takes over, eliminating single points of failure in the scheduling process and ensuring uninterrupted task scheduling, thus improving system high availability. Each computing node 120 deploys a daemon process 121, continuously monitoring node resources and worker process status. When a worker process exits, a new process is automatically started to ensure continuous task execution, manage node resource status, and provide accurate resource data support for the scheduling process 110. The streaming and inference processes 1222 transmit video frame metadata via a shared memory circular buffer frame queue; pixel data is transmitted via shared memory or shared video memory. Transmitting pixel data via shared video memory incurs no copying cost, significantly improving inference efficiency. The circular buffer frame queue efficiently transmits metadata, reducing inter-process communication overhead. Information and task status data are registered using storage nodes such as etcd, enabling efficient sharing of status data between processes and supporting rapid task recovery during scheduling master / slave failover and worker process restarts. Scheduling, daemon, and worker processes use sockets to perform tasks for adding, deleting, modifying, and querying, improving system response speed. Through the synergistic effect of these technologies, a multi-layered fault-tolerance mechanism is formed, ensuring continuous system operation even when processes are abnormal. Overall efficiency is improved in terms of resource reuse, data transmission, and hardware utilization. The layered architecture allows each process to expand independently, adapting to business growth needs, and collectively achieving high availability, efficient resource utilization, and fault tolerance, solving the problems of resource waste, poor stability, and low hardware utilization in existing technologies.

[0096] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the storage medium embodiments and computer device embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0097] This application can be used in a wide range of general-purpose or specialized in-vehicle computing system environments or configurations. Examples include: personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative; multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multi-process task inference system, characterized in that, The multi-process task inference system includes a scheduling process and multiple computing nodes; The scheduling process responds to external task inference requests by parsing the tasks and querying the current working status of each computing node. The scheduling process selects the computing power node to perform the task inference based on the parsing results of the task and the current working status of each computing power node. The computing node performs the inference of the task and sends the inference result of the task to the scheduling process after execution to update the processing status of the task.

2. The multi-process task inference system according to claim 1, characterized in that, Each of the aforementioned computing nodes includes a daemon process and multiple processing units; The daemon process is used to receive query requests sent by the scheduling process, obtain the working status of each processing unit, so as to obtain the current working status of the computing power node to which it belongs, and automatically generate a new processing unit when the processing unit exits the running state.

3. The multi-process task inference system according to claim 2, characterized in that, The processing unit includes a stream fetching process and an inference process; The streaming process retrieves streaming data from an external video source according to the instructions contained in the task, and converts the streaming data into raw video frames; The inference process performs the inference for the task through an algorithm plugin.

4. The multi-process task inference system according to claim 3, characterized in that, The computing node also includes a node shared memory area, through which the processing units communicate with each other.

5. The multi-process task inference system according to claim 4, characterized in that, The processing unit also includes a shared video memory region, through which the streaming process and the inference process share pixel data of the original video frame.

6. The multi-process task inference system according to claim 4, characterized in that, The node shared memory region includes a frame queue consisting of a circular buffer. The fetching process puts the pixel data into the frame queue, and the inference process retrieves the pixel data from the frame queue through real-time consumption.

7. The multi-process task inference system according to claim 3, characterized in that, The stream-fetching process, the inference process, and the daemon process communicate directly via sockets, and the daemon process and the scheduling process communicate directly via sockets.

8. The multi-process task inference system according to claim 3, characterized in that, The inference process includes multiple types of algorithm plugins. Based on the requirements of the task, the inference process calls the corresponding algorithm plugins to perform the inference for the task.

9. The multi-process task inference system according to claim 1, characterized in that, The multi-process task inference system is deployed with at least two scheduling processes, and the scheduling processes form a master-slave relationship. When the primary scheduling process exits abnormally, the backup scheduling process automatically takes over the task processing progress of the primary scheduling process and becomes the new primary scheduling process.

10. The multi-process task inference system according to claim 1, characterized in that, The node registration information and task status data of the computing nodes are stored in a third-party key-value storage service.

11. A multi-process task reasoning method, characterized in that, The multi-process task inference method is used in the multi-process task inference system as described in any one of claims 1-10, comprising: In response to external task inference requests, query the current working status of each computing node; Based on the query results and the reasoning requirements of the task, a computing power node is selected to reason about the task. After the reasoning for the task is completed, the reasoning result of the task is fed back to the outside.

12. A computer device, characterized in that, The computer device is equipped with a multi-process task inference system as described in any one of claims 1-10.

Images