Efficient deep learning model deployment scheme based on message queue and micro-service architecture

Abstract

The invention provides an efficient deep learning model deployment scheme, and high-concurrency request processing and batch reasoning of large-scale image processing are realized by adopting a message queue and a micro-service architecture. The system comprises a front-end calling party, a SprBoot cluster, a RabbitMQ message queue, a PyTorch-based algorithm instance and a Redis cache, and matching of a plurality of components aims at achieving efficient request processing, batch image reasoning and rapid result query. The scheme can significantly improve the processing capability and response speed of the system, and is suitable for large-scale real-time image processing application. Compared with a traditional stand-alone processing mode which often shows the problems of low efficiency and long response time when facing high-concurrency requests, the scheme can effectively improve the throughput of the cluster, reduce the detection delay, and finally greatly improve the real-time detection capability of the cluster.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically designs an efficient deep learning model deployment solution based on message queue and microservice architecture. Background Art

[0002] In modern intelligent monitoring systems, a large number of cameras are usually deployed for real-time monitoring and detection of abnormal situations. Each camera needs to continuously send detection requests to the backend server and require the server to process these requests in real time to determine whether there are abnormalities in the current scene. However, the traditional single-machine processing method often shows problems of low efficiency and long response time when facing high concurrent requests, which are specifically manifested in the following aspects:

[0003] Low processing efficiency: In traditional methods, every time the backend server receives a request, it directly calls the deep learning model for inference. This one-by-one processing method makes the deep learning model always process a single image, making it difficult to utilize the parallel computing capabilities of modern hardware, resulting in low processing efficiency.

[0004] Insufficient system response capability: In a large-scale deployment environment, a large number of cameras send detection requests at the same time. Traditional methods are difficult to process all requests in a short period of time, which can easily cause system overload, resulting in request backlogs, significantly prolonged response time, and even request loss.

[0005] Uneven resource utilization: In a high-concurrency environment, some servers may be overloaded due to processing a large number of requests, while other servers may be idle, unable to fully utilize system resources, reducing overall system performance.

[0006] In order to solve the above problems, the present invention proposes an efficient deep learning model deployment solution based on distributed computing and microservice architecture, which realizes efficient request processing and batch image reasoning by rationally utilizing message queues and cache systems. Summary of the invention

[0007] In order to deal with the problems of low efficiency and long response time of traditional single-machine processing methods when facing high concurrent requests, this paper proposes an efficient deep learning model deployment solution based on message queue and microservice architecture, which aims to achieve efficient request processing, batch image reasoning and fast result query by introducing SpringBoot cluster, RabbitMQ message queue, PyTorch-based algorithm instance and Redis cache system. This solution can significantly improve the processing power and response speed of the system, and is suitable for large-scale real-time image processing applications.

[0008] Specific content:

[0009] An efficient deep learning model deployment method based on message queue and microservice architecture includes the following steps:

[0010] Step 1: The front-end camera captures video frames in real time and extracts image data. The camera sends a detection request to the SpringBoot instance, which contains the image data to be detected.

[0011] Step 2: After receiving the request, the SpringBoot instance stores the image data in the RabbitMQ message queue and immediately returns a successful response containing msgID to the caller, notifying it that the request has been successfully received.

[0012] Step 3: The algorithm model deployed on the high-performance computing node reads image data in batches from the RabbitMQ message queue. After preprocessing, the batch-read image data is input into the deep learning model for inference. The inference process includes image preprocessing, model inference, and result post-processing to ensure the accuracy and timeliness of the detection results.

[0013] Step 4: The inference result is stored in the Redis cache system with msgID as the key and the detection result as the value. The Redis cache system supports high-concurrency read and write operations, and can quickly store and retrieve a large number of inference results to ensure efficient queries;

[0014] Step 5: The front-end caller uses msgID as the query condition and obtains the corresponding detection results from the Redis cache system by calling the callback interface of the SpringBoot backend.

[0015] Step 1 of the present invention comprises:

[0016] Step 1-1, the front-end camera captures video frames in real time and extracts image data; specifically, each camera captures video data at a fixed frame rate, and performs preliminary processing on the video data through the embedded system to extract static images. Cameras usually have certain pre-processing capabilities, such as image compression, noise filtering, etc., to ensure the quality of transmitted image data;

[0017] Step 1-2: The camera sends a detection request to the SpringBoot instance, which contains the image data to be detected. The camera sends the extracted image data to the SpringBoot instance through a network interface (such as Wi-Fi, Ethernet). The detection request usually contains metadata of the image, such as timestamp, camera ID, image format, etc., for subsequent processing and tracking;

[0018] Step 2 of the present invention comprises:

[0019] Step 2-1: After receiving the request, the SpringBoot instance stores the image data in the RabbitMQ message queue. The SpringBoot instance receives the image data through the REST API, encapsulates the data into messages, and pushes them to the RabbitMQ message queue. In addition to the image data, each message also comes with a unique msgID for identification and tracking;

[0020] Step 2-2, the SpringBoot instance immediately returns a successful response containing msgID to the caller, notifying it that the request has been successfully received; the returned response contains not only msgID, but may also include information such as receiving time and estimated processing time, so that the caller can manage the request;

[0021] Step 3 of the present invention comprises:

[0022] Step 3-1: The algorithm model deployed on the high-performance computing node reads image data in batches from the RabbitMQ message queue. The algorithm model periodically pulls a certain amount of image data from the message queue in batches through the RabbitMQ client to improve processing efficiency. Batch reading can be dynamically adjusted according to the queue length and system load to ensure optimal utilization of system resources.

[0023] Step 3-2: The batch-read image data is preprocessed and then input into the deep learning model for inference; the preprocessing steps include image scaling, normalization, data enhancement, etc. to meet the input requirements of the deep learning model. The preprocessed image data is input into the deep learning model in batches for feature extraction and classification;

[0024] Step 3-3, the reasoning process includes image preprocessing, model reasoning and result post-processing to ensure the accuracy and timeliness of the detection results; during the reasoning process, the model will generate intermediate feature maps, which can be used for further analysis and debugging. Result post-processing includes non-maximum suppression (NMS), threshold filtering, etc. to ensure the accuracy of the detection results;

[0025] Step 4 of the present invention comprises:

[0026] Step 4-1, the inference result is stored in the Redis cache system with msgID as the key and the detection result as the value; each inference result will be accompanied by msgID for subsequent query and retrieval. The Redis cache system has a persistence mechanism to ensure that data will not be lost when the system restarts or fails;

[0027] Step 4-2, the Redis cache system supports high-concurrency read and write operations, and can quickly store and retrieve a large number of inference results to ensure the efficiency of queries; Redis uses memory storage, with extremely fast read and write speeds, which is suitable for fast response requirements in high-concurrency environments. The cache system supports data expiration policies, which can automatically clean up expired data and optimize storage resources;

[0028] Step 5 of the present invention comprises:

[0029] Step 5-1: The front-end caller uses msgID as the query condition and obtains the corresponding detection results from the Redis cache system by calling the callback interface of the SpringBoot backend. The front-end caller uses msgID to query the detection results regularly or according to user operations to obtain the latest detection information. The query request is sent through the API interface of the SpringBoot backend, and the backend retrieves the corresponding detection results from the Redis cache system.

[0030] Step 5-2, the SpringBoot backend queries the Redis cache system and returns the detection results to the frontend caller; the backend processes the query request, accesses the Redis cache system, and obtains the detection results corresponding to the msgID. The returned detection results include detailed information such as detection type, confidence, and detection location, which is convenient for the frontend caller to display and further process;

[0031] Step 5-3: After receiving the test results, the front-end caller can take corresponding measures based on the results, such as issuing an alarm or further processing; after receiving the test results, the front-end system can make judgments and processes according to preset rules, such as issuing an alarm, recording a log, or initiating emergency measures. The test results can also be used to generate statistical reports to help users analyze and optimize the monitoring system.

[0032] The present invention combines message queues and caches to cope with real-time monitoring in high-concurrency scenarios. Compared with the existing stand-alone architecture, the method proposed in the present invention can significantly improve the processing power and response speed of the system during detection, and is suitable for large-scale real-time image processing applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the overall architecture diagram of the present invention;

[0034] Figure 2 It is the overall flow chart of the present invention;

[0035] Figure 3 It is the SpringBoot interaction diagram; DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0037] The present invention is implemented including:

[0038] The overall structure of the method proposed by the present invention is as follows: Figure 1 As shown, the overall process is as follows Figure 2 As shown, SpringBoot interacts with other components as follows Figure 3 As shown, the steps for constructing the deployment scheme in the embodiment of the present invention are as follows:

[0039] Step 1: The front-end camera captures video frames in real time and extracts image data. The camera sends a detection request to the SpringBoot instance. The request contains the image data to be detected. This step also requires Nginx to distribute the request to the specific instance of the SpringBoot cluster.

[0040] Step 2: After receiving the request, the SpringBoot instance stores the image data in the RabbitMQ message queue and immediately returns a successful response containing msgID to the caller, notifying it that the request has been successfully received.

[0041] Step 3: The algorithm model deployed on the high-performance computing node reads image data in batches from the RabbitMQ message queue. After preprocessing, the batch-read image data is input into the deep learning model for inference. The inference process includes image preprocessing, model inference, and result post-processing to ensure the accuracy and timeliness of the detection results.

[0042] Step 4: The inference result is stored in the Redis cache system with msgID as the key and the test result as the value. The Redis cache system supports high-concurrency read and write operations, and can quickly store and retrieve a large number of inference results to ensure the efficiency of queries. In addition, Redis also supports persistence operations to facilitate result backtracking;

[0043] Step 5: The front-end caller uses msgID as the query condition and obtains the corresponding detection results from the Redis cache system by calling the callback interface of the SpringBoot backend.

[0044] Step 1 of the present invention comprises:

[0045] Step 1-1, the front-end camera captures video frames in real time and extracts image data; specifically, each camera captures video data at a fixed rate, and then packages the data into a request and sends it to the gateway;

[0046] Step 1-2: The camera sends a detection request to the SpringBoot instance, which contains the image data to be detected. The camera sends the extracted image data to the SpringBoot instance through a network interface (such as Wi-Fi, Ethernet). The detection request usually contains metadata of the image, such as timestamp, camera ID, image format, etc., for subsequent processing and tracking;

[0047] Step 2 of the present invention comprises:

[0048] Step 2-1: After receiving the request, the SpringBoot instance stores the image data in the RabbitMQ message queue. The SpringBoot instance receives the image data through the REST API, encapsulates the data into messages, and pushes them to the RabbitMQ message queue. In addition to the image data, each message also comes with a unique msgID for identification and tracking;

[0049] Step 2-2, the SpringBoot instance immediately returns a successful response containing msgID to the caller, notifying it that the request has been successfully received; the returned response contains not only msgID, but may also include information such as receiving time and estimated processing time, so that the caller can manage the request;

[0050] Step 3 of the present invention comprises:

[0051] Step 3-1: The algorithm model deployed on the high-performance computing node reads image data in batches from the RabbitMQ message queue. The algorithm model periodically pulls a certain amount of image data from the message queue in batches through the RabbitMQ client to improve processing efficiency. Batch reading can be dynamically adjusted according to the queue length and system load to ensure optimal utilization of system resources.

[0052] Step 3-2: The batch-read image data is preprocessed and then input into the deep learning model for inference; the preprocessing steps include image scaling, normalization, data enhancement, etc. to meet the input requirements of the deep learning model. The preprocessed image data is input into the deep learning model in batches for feature extraction and classification;

[0053] Step 3-3, the reasoning process includes image preprocessing, model reasoning and result post-processing to ensure the accuracy and timeliness of the detection results; during the reasoning process, the model will generate intermediate feature maps, which can be used for further analysis and debugging. Result post-processing includes non-maximum suppression (NMS), threshold filtering, etc. to ensure the accuracy of the detection results;

[0054] Step 4 of the present invention comprises:

[0055] Step 4-1, the inference result is stored in the Redis cache system with msgID as the key and the detection result as the value; each inference result will be accompanied by msgID for subsequent query and retrieval. The Redis cache system has a persistence mechanism to ensure that data will not be lost when the system restarts or fails;

[0056] Step 4-2, the Redis cache system supports high-concurrency read and write operations, and can quickly store and retrieve a large number of inference results to ensure the efficiency of queries; Redis uses memory storage, with extremely fast read and write speeds, which is suitable for fast response requirements in high-concurrency environments. The cache system supports data expiration policies, which can automatically clean up expired data and optimize storage resources;

[0057] Step 5 of the present invention comprises:

[0058] Step 5-1: The front-end caller uses msgID as the query condition and obtains the corresponding detection results from the Redis cache system by calling the callback interface of the SpringBoot backend. The front-end caller uses msgID to query the detection results regularly or according to user operations to obtain the latest detection information. The query request is sent through the API interface of the SpringBoot backend, and the backend retrieves the corresponding detection results from the Redis cache system.

[0059] Step 5-2, the SpringBoot backend queries the Redis cache system and returns the detection results to the frontend caller; the backend processes the query request, accesses the Redis cache system, and obtains the detection results corresponding to the msgID. The returned detection results include detailed information such as detection type, confidence, and detection location, which is convenient for the frontend caller to display and further process;

[0060] Step 5-3: After receiving the test results, the front-end caller can take corresponding measures based on the results, such as issuing an alarm or further processing; after receiving the test results, the front-end system can make judgments and processes according to preset rules, such as issuing an alarm, recording a log, or initiating emergency measures. The test results can also be used to generate statistical reports to help users analyze and optimize the monitoring system.

[0061] The present invention combines message queues and caches to cope with real-time monitoring in high-concurrency scenarios. Compared with the existing stand-alone architecture, the method proposed in the present invention can significantly improve the processing power and response speed of the system during detection, and is suitable for large-scale real-time image processing applications.

[0062] The present invention provides an efficient deep learning model deployment solution based on message queues and microservice architecture. The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An efficient deep learning model deployment solution based on message queue and microservice architecture, characterized in that: The following steps are involved: Step 1: The front-end camera captures video frames in real time and extracts image data. The camera sends a detection request to the SpringBoot instance, which contains the image data to be detected. Step 2: After receiving the request, the SpringBoot instance stores the image data in the RabbitMQ message queue and immediately returns a successful response containing msgID to the caller, notifying it that the request has been successfully received. Step 3: The algorithm model deployed on the high-performance computing node reads image data in batches from the RabbitMQ message queue. After preprocessing, the batch-read image data is input into the deep learning model for inference. The inference process includes image preprocessing, model inference, and result post-processing to ensure the accuracy and timeliness of the detection results. Step 4: The inference result is stored in the Redis cache system with msgID as the key and the detection result as the value. The Redis cache system supports high-concurrency read and write operations, and can quickly store and retrieve a large number of inference results to ensure the efficiency of queries. ; Step 5: The front-end caller uses msgID as the query condition and obtains the corresponding detection results from the Redis cache system by calling the callback interface of the SpringBoot backend.

2. According to the method of claim 1, step 2 of the present invention comprises: Step 2-1: After receiving the request, the SpringBoot instance stores the image data in the RabbitMQ message queue. The SpringBoot instance receives the image data through the REST API, encapsulates the data into messages, and pushes them to the RabbitMQ message queue. In addition to the image data, each message also comes with a unique msgID for identification and tracking; In step 2-2, the SpringBoot instance immediately returns a successful response containing msgID to the caller, notifying it that the request has been successfully received; the returned response not only contains msgID, but may also include information such as receiving time and estimated processing time, to facilitate the caller to manage the request.

3. According to the method of claim 1, step 3 of the present invention comprises: Step 3-1: The algorithm model deployed on the high-performance computing node reads image data in batches from the RabbitMQ message queue. The algorithm model periodically pulls a certain amount of image data from the message queue in batches through the RabbitMQ client to improve processing efficiency. Batch reading can be dynamically adjusted according to the queue length and system load to ensure optimal utilization of system resources. Step 3-2: The batch-read image data is preprocessed and then input into the deep learning model for inference; The preprocessing steps include image scaling, normalization, data enhancement, etc. to meet the input requirements of the deep learning model. The preprocessed image data is input into the deep learning model in batches for feature extraction and classification; Step 3-3, the reasoning process includes image preprocessing, model reasoning and result post-processing to ensure the accuracy and timeliness of the detection results; during the reasoning process, the model will generate intermediate feature maps, which can be used for further analysis and debugging. Result post-processing includes non-maximum suppression (NMS), threshold filtering, etc. to ensure the accuracy of the detection results.

4. According to the method of claim 1, step 4 of the present invention comprises: Step 4-1, the inference result is stored in the Redis cache system with msgID as the key and the detection result as the value; Each inference result is accompanied by a msgID to facilitate subsequent query and retrieval. The Redis cache system has a persistence mechanism to ensure that data is not lost when the system restarts or fails; Step 4-2: The Redis cache system supports high-concurrency read and write operations, and can quickly store and retrieve a large number of inference results to ensure the efficiency of queries. Redis uses memory storage, which has extremely fast read and write speeds and is suitable for fast response requirements in high-concurrency environments. The cache system supports data expiration policies, which can automatically clean up expired data and optimize storage resources.

5. According to the method of claim 1, step 5 of the present invention comprises: Step 5-1: The front-end caller uses msgID as the query condition and obtains the corresponding detection results from the Redis cache system by calling the callback interface of the SpringBoot backend. The front-end caller uses msgID to query the detection results regularly or according to user operations to obtain the latest detection information. The query request is sent through the API interface of the SpringBoot backend, and the backend retrieves the corresponding detection results from the Redis cache system. Step 5-2, the SpringBoot backend queries the Redis cache system and returns the detection results to the frontend caller; the backend processes the query request, accesses the Redis cache system, and obtains the detection results corresponding to the msgID. The returned detection results include detailed information such as detection type, confidence, and detection location, which is convenient for the frontend caller to display and further process; Step 5-3: After receiving the detection results, the front-end caller can take corresponding measures based on the results, such as issuing an alarm or performing further processing; After receiving the test results, the front-end system can make judgments and processes according to preset rules, such as issuing alarms, recording logs, or initiating emergency measures. The test results can also be used to generate statistical reports to help users analyze and optimize the monitoring system.

Images