Adaptive reasoning optimization method based on target detection model in limited environment

Abstract

The invention relates to an adaptive reasoning optimization method based on a target detection model in a limited environment, and belongs to the field of video target detection. According to the invention, under the environment that the CPU, the NPU and the memory resources are limited, the process of target detection processing and display of the video is completed, the video is disassembled into image frames, the image frames are distributed to a plurality of processes according to the frame sequence, and the processing of each frame of image is completed. Meanwhile, the invention provides a method for optimizing the total number n of processes in a limited environment, a method for calculating the FPS, a frame rate shaping output method and a method for setting the length of a multi-process processing queue. By optimizing process management, computing resource allocation and queue length setting, efficient and stable video target detection is realized on an embedded device or a mobile platform with limited resources.

Description

Technical Field

[0001] The present invention belongs to the field of video target detection, and in particular relates to an adaptive reasoning optimization method based on a target detection model in a restricted environment. Background Art

[0002] With the rapid development of intelligent video analysis technology, object detection has become a key technology in the fields of video surveillance, autonomous driving, and intelligent interaction. However, these application scenarios often face the challenge of limited resources, especially on embedded devices and mobile platforms, where CPU, NPU (neural network processing unit) and memory resources are relatively limited. In this case, how to efficiently use limited computing resources to ensure the real-time and accuracy of video object detection has become an urgent problem to be solved. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] The technical problem to be solved by the present invention is how to provide an adaptive reasoning optimization method based on a target detection model in a constrained environment to solve the problem of efficiently utilizing limited computing resources and ensuring the real-time and accuracy of video target detection.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention proposes an adaptive reasoning optimization method based on a target detection model in a constrained environment, the method comprising:

[0007] Assume that the total number of processes is n, and each process includes three parts: preprocessing, reasoning, and postprocessing. The CPU completes the preprocessing and postprocessing processes, and the NPU completes the reasoning process.

[0008] First, by dividing the video into frames, the 1st frame to the nth frame are distributed to the corresponding n processes in sequence, and then the n+1th frame to the 2nth frame are distributed to the corresponding processes in sequence, and so on;

[0009] For any process i, a queue A is used i Save the original video information. The queue saves the information of the i-th frame, the n+i-th frame, the 2n+i-th frame, etc. in order. The message format in the queue is [frame_id, frame, start_time], which respectively represent the frame id, frame content and start time. The maximum queue length is a. start_time records the start time when the frame enters the process and is processed.

[0010] The queue content is subjected to preprocessing, reasoning and postprocessing in turn to obtain the result queue Bi. The message format in Bi is [frame_id, result, finish_time], which respectively represents the frame id, result and end time. The maximum queue length is b, and finish_time records the end time of the process processing completion.

[0011] The display node sequentially takes out the results from process 1 to process n and displays them, thereby ensuring that the results are displayed in the correct order.

[0012] (III) Beneficial effects

[0013] The present invention proposes an adaptive reasoning optimization method based on a target detection model in a constrained environment. The present invention optimizes process management, computing resource allocation, and queue length setting to achieve efficient and stable video target detection on resource-constrained embedded devices or mobile platforms, thereby promoting the application of intelligent video analysis technology in more scenarios. The present invention has the following beneficial effects:

[0014] 1. Improve the real-time performance of video target detection and ensure that a high frame processing rate can be maintained in a restricted environment.

[0015] 2. Optimize CPU and NPU resource utilization to avoid resource waste.

[0016] 3. Ensure the smoothness of video output and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A process diagram of object detection processing and display of a video of the present invention;

[0018] Figure 2 Schematic diagram for stable real FPS determination. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] In order to solve the above problems, the present invention proposes a method for optimizing video adaptive reasoning based on target detection in an environment with limited CPU, NPU and memory resources. The method aims to achieve the following goals by optimizing process management, computing resource allocation and queue length setting:

[0021] 1. Improve the real-time performance of video target detection and ensure that a high frame processing rate can be maintained in a restricted environment.

[0022] 2. Optimize CPU and NPU resource utilization to avoid resource waste.

[0023] 3. Ensure the smoothness of video output and improve user experience.

[0024] Through the above method, it is expected that efficient and stable video target detection can be achieved on resource-constrained embedded devices or mobile platforms, thereby promoting the application of intelligent video analysis technology in more scenarios. The present invention will provide a new idea and practical reference for the optimization of video target detection in a constrained environment.

[0025] In an environment with limited CPU, NPU and memory resources, the process of object detection, processing and display of the video is completed. The specific method is as follows Figure 1 The basic idea is to decompose the video into image frames, distribute the image frames to multiple processes in frame order, and complete the processing of each frame.

[0026] The present invention proposes an adaptive reasoning optimization method based on a target detection model in a constrained environment. In a resource-constrained environment, the specific process of using multi-process reasoning video is as follows:

[0027] Assume that the total number of processes is n, and each process includes three parts: preprocessing, reasoning, and postprocessing. The CPU completes the preprocessing and postprocessing processes, and the NPU completes the reasoning process.

[0028] First, by dividing the video into frames, the 1st frame to the nth frame are distributed to the corresponding n processes in sequence, and then the n+1th frame to the 2nth frame are distributed to the corresponding processes in sequence, and so on;

[0029] For any process i, a queue A is used i Save the original video information. The queue saves the information of the i-th frame, the n+i-th frame, the 2n+i-th frame, etc. in order. The message format in the queue is [frame_id, frame, start_time], which respectively represent the frame id, frame content and start time. The maximum queue length is a. start_time records the start time when the frame enters the process and is processed.

[0030] The queue contents are processed through preprocessing, reasoning and postprocessing in sequence to obtain the result queue B i , B i The message format is [frame_id, result, finish_time], which represent frame id, result and end time respectively. The maximum queue length is b, and finish_time records the end time of process processing completion.

[0031] The display node sequentially takes out the results from process 1 to process n and displays them, thereby ensuring that the results are displayed in the correct order.

[0032] in,

[0033] In order to measure the performance of this method, the FPS evaluation index is used. FPS (Frames Per Second) refers to the number of frames processed per second. The FPS value calculated in real time changes dramatically and is difficult to reflect the system performance. The present invention proposes to calculate the real FPS in a sliding window manner to evaluate the performance. The window size T1 is the maximum end-to-end inference time, and the formula is That is, the inference time of the frame with the longest inference time among all frames processed by all processes. The window update interval T2 is the minimum end-to-end inference time, and T2 is calculated as follows: That is, the inference time of the frame with the shortest inference time for each frame processed by all processes. It should be noted that in the post-processing stage of process processing, since the processing time of NMS (Non-Maximum Suppression) is related to the number of detection results, the common practice is to set the maximum time threshold p. When the NMS processing time exceeds p seconds, stop the calculation and exit to prevent the T1 value caused by NMS from increasing infinitely. The p value should be set as a hyperparameter according to the target task type. Assuming that m frames are processed within the T1 time period (T1 is defined as the inference time of the frame with the longest inference time for each frame processed by all processes. Note that it is the longest inference time of the frame, but most frames do not take the longest inference time. Therefore, at least one frame is processed within this inference time period. In more cases, multiple frames are processed, so it is assumed that it processes m frames), the calculation formula for FPS is The FPS update interval is T2.

[0034] Assuming that the total number of processes n has been adjusted according to the restricted environment, the FPS tends to stabilize at the maximum value, but the real FPS still has a small range of variation. In order to make the output picture smooth, the present invention proposes a video frame rate shaping mechanism, that is, setting an output FPS frame rate to ensure the stability of the video output frame rate.

[0035] In order to make full use of the system's computing resources, the total number of processes n, the maximum length a of the input queue Ai, and the maximum length b of the result queue Bi need to be adjusted according to the restricted environment, and the output FPS frame rate needs to be calculated based on the actual FPS. The specific adjustment methods for the above parameters are as follows:

[0036] In a stable restricted environment, the real FPS tends to be stable, so a can be set equal to b. The setting formula of b is It can effectively utilize resources while taking into account speed and saving memory.

[0037] For the total number of processes n, the most suitable n value is different for any different restricted environment. However, the method for adjusting n described in the present invention is applicable to any restricted environment. The most suitable n value corresponds to the maximum output FPS frame rate and the n value only depends on the CPU and NPU computing resources of the restricted environment. When the n value is dynamically adjusted, the adjustment of the input queue is too complicated to reflect the result through FPS. Therefore, each time the n value is adjusted, all the created A i and B i Queue and restart the FPS calculation.

[0038] To find the appropriate n value: refer to the common congestion control method AIMD (Additive Increase Multiplicative Decrease) in the TCP protocol, and use the Slow start algorithm to increase the initial value of n, that is, use an exponential method to double the 2n value each time. When the real FPS begins to decrease, use a multiplicative reduction method for the n value, multiply the current n value by a coefficient of 0.5, and convert it to an integer value. When the real FPS continues to increase, use an additive increase method, and increase the n value by 1 each time. The multiplicative reduction and additive increase methods are repeated to obtain a stable real FPS maximum value. The schematic diagram is as follows Figure 2 shown.

[0039] Since the real FPS changes dynamically, it is necessary to set an interval as the stable state of FPS and prove that n is the most reasonable value. Assuming that the distribution of video data conforms to the normal distribution, the FPS also conforms to the normal distribution. When the average value of FPS in the stable state output of a certain n value is greater than 95.45% of the maximum value (within two standard deviations), and the average value of this FPS is the largest, then this n value is the optimal value of the current restricted environment, and the average FPS value obtained is the stable FPS value. The b value can be further calculated by the setting formula of b, and the a value can be calculated.

[0040] In order to make the output picture smooth, the present invention proposes a video frame rate shaping mechanism, that is, setting an output FPS frame rate to ensure the stability of the video output frame rate. Assuming that the stable value of FPS conforms to the normal distribution, in order to make the output picture stable, the output FPS frame rate is set to:

[0041] Output FPS = stable FPS × 95.45%;

[0042] This ensures the smoothness of the output image.

[0043] The main innovative features of the present invention include:

[0044] 1. Propose a method to optimize the total number of processes n for a constrained environment.

[0045] 2. Propose a method for calculating FPS and a frame rate shaping output method for constrained environments.

[0046] 3. Propose a method for setting the length of multi-process processing queues for constrained environments.

[0047] The present invention achieves efficient and stable video target detection on resource-constrained embedded devices or mobile platforms by optimizing process management, computing resource allocation and queue length setting, thereby promoting the application of intelligent video analysis technology in more scenarios. The present invention has the following beneficial effects:

[0048] 1. Improve the real-time performance of video target detection and ensure that a high frame processing rate can be maintained in a restricted environment.

[0049] 2. Optimize CPU and NPU resource utilization to avoid resource waste.

[0050] 3. Ensure the smoothness of video output and improve user experience.

[0051] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An adaptive reasoning optimization method based on a target detection model in a constrained environment, characterized in that: The method includes: Assume that the total number of processes is n, and each process includes three parts: preprocessing, reasoning, and postprocessing. The CPU completes the preprocessing and postprocessing processes, and the NPU completes the reasoning process. First, by dividing the video into frames, the 1st frame to the nth frame are distributed to the corresponding n processes in sequence, and then the n+1th frame to the 2nth frame are distributed to the corresponding processes in sequence, and so on; For any process i, a queue A is used i Save the original video information. The queue saves the information of the i-th frame, the n+i-th frame, the 2n+i-th frame, etc. in order. The message format in the queue is [frame_id, frame, start_time], which respectively represent the frame id, frame content and start time. The maximum queue length is a. start_time records the start time when the frame enters the process and is processed. The queue contents are processed through preprocessing, reasoning and postprocessing in sequence to obtain the result queue B i , B i The message format is [frame_id, result, finish_time], which represent frame id, result and end time respectively. The maximum queue length is b, and finish_time records the end time of process processing completion. The display node sequentially takes out the results from process 1 to process n and displays them, thereby ensuring that the results are displayed in the correct order.

2. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 1, characterized in that: The actual FPS is calculated using a sliding window to evaluate the performance; the window size T1 is the maximum end-to-end inference time, and the formula is: That is, the inference time of the frame with the longest inference time among all frames processed by all processes; the window update interval T2 is the minimum end-to-end inference time, and T2 is calculated as follows: That is, the inference time of the frame with the shortest inference time among all frames processed by all processes; assuming that m frames are processed within the T1 time period, the calculation formula for FPS is The time interval for FPS update is T2.

3. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 2, characterized in that: In the post-processing stage of process processing, since the processing time of non-maximum suppression NMS is related to the number of detection results, a maximum time threshold p is set. When the NMS processing time exceeds p seconds, the calculation is stopped and exited to prevent the T1 value caused by NMS from increasing infinitely.

4. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 3 is characterized in that: The p-value is set as a hyperparameter based on the target task type.

5. The adaptive reasoning optimization method based on the target detection model in a constrained environment according to any one of claims 1 to 4, characterized in that: This method adopts the video frame rate shaping mechanism, that is, setting an output FPS frame rate to ensure the stability of the video output frame rate; among them, the total number of processes n and the input queue A need to be adjusted according to the restricted environment. i The maximum length a and the result queue B i The maximum length b, the output FPS frame rate needs to be calculated based on the actual FPS.

6. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 5, characterized in that: The most appropriate n value corresponds to the maximum output FPS frame rate and the n value depends only on the CPU and NPU computing resources of the limited environment; each time the n value is adjusted, all the created A i and B i Queue and restart the FPS calculation.

7. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 5, characterized in that: When finding a suitable n value: use the Slow start algorithm to increase the initial value of n, that is, use an exponential method to double the 2n value each time; when the real FPS starts to decrease, use a multiplicative reduction method for the n value, multiply the current n value by a coefficient of 0.5, and convert it to an integer value; when the real FPS continues to increase, use an additive increase method, increasing the n value by 1 each time; repeat the multiplicative reduction and additive increase methods to obtain a stable real FPS maximum value; Assuming that the distribution of video data conforms to the normal distribution, the FPS also conforms to the normal distribution. When the average FPS value in the stable state output of a certain n value is greater than 95.45% of the maximum value, and this FPS average value is the largest, then this n value is the optimal value of the current restricted environment, and the FPS average value obtained is the stable FPS value.

8. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 7, characterized in that: In a stable restricted environment, the real FPS tends to be stable, so set a equal to b.

9. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 8, characterized in that: The setting formula for b is 10. The adaptive reasoning optimization method based on the target detection model in a constrained environment as claimed in claim 8, characterized in that: Set an output FPS frame rate to ensure the stability of the video output frame rate. Set the output FPS frame rate to: Output FPS = stable FPS × 95.45%.

Images