Image processing performance optimization method, system, device and storage medium

Abstract

The present invention discloses an image processing performance optimization method, system, device and storage medium, which are applied to the field of image processing technology, including: constructing a unified data structure based on the data dependency relationship between operators in each frame, and using a dynamic programming method to determine the queue order of operators in the previous and next frames; based on the queue order, implementing a scheduler, when the operator on which the head element of the execution queue in the scheduler depends is executed, the scheduler dequeues the execution queue and distributes the dequeued operator to the execution thread of the corresponding computing resource for execution. The present invention effectively improves the utilization rate of heterogeneous computing resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing performance optimization method, system, device and storage medium. Background Art

[0002] Current smart cameras, security cameras, and even mobile phones often have more than one type of computing chip, such as NPU, DSP, GPU, etc. Using these chips to collaborate for computing is called heterogeneous computing. In theory, these chips can run at the same time to maximize computing power.

[0003] However, in the traditional image processing process, it is usually performed frame by frame, and each frame needs to call multiple different operators. These operators usually run on different chips on heterogeneous devices. In the process of frame-by-frame calculation, since the previous and next operators in the same frame often have data dependencies, for example, the input of the next operator is the output of the previous operator, the next operator must wait until the previous operator is completed before it can have input content. This data dependency relationship makes heterogeneous chips only run one at a time, thus wasting most of the computing power, making it often impossible for devices such as cameras to achieve an ideal frame rate.

[0004] Therefore, how to provide an image processing performance optimization method, system, device and storage medium that can overcome the problem that only one heterogeneous chip can be run at the same time due to the data dependency brought about by frame-by-frame processing, resulting in a waste of most computing power, and effectively improve the utilization rate of heterogeneous computing resources is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention proposes an image processing performance optimization method, system, device and storage medium. Based on the fact that most of the data of the next frame is independent of the data of the previous frame, it is proposed to change from frame-by-frame processing to "delay one frame" processing. When the frame rate that can be achieved by the image processing process running frame by frame is less than the set frame rate, the image processing process directly obtains the next frame first, and performs image processing at the same time as the image that has not yet been output. For example, the last few operators of the image processing process are calculated using DSP, while the first few are calculated using NPU. When the image processing of the previous and next frames is performed simultaneously, the previous frame uses DSP while the next frame uses NPU, which greatly improves the utilization rate of heterogeneous computing resources.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A method for optimizing image processing performance, comprising:

[0008] Step 1: Based on the data dependency between operators in each frame, a unified data structure is constructed, and the order in which operators of the previous and next frames are queued is determined using dynamic programming.

[0009] Step 2: Based on the order of enqueuing, implement a scheduler. When the operator that the head element of the execution queue in the scheduler depends on is executed, the scheduler will dequeue the execution queue and distribute the dequeued operator to the execution thread of the corresponding computing resource for execution.

[0010] Optionally, in step 1, a dynamic programming method is used to determine the order in which operators of the previous and next frames are queued, specifically:

[0011] Each operator in the data structure is taken as a truncation point, and the dynamic programming method is used to determine the optimal operator enqueue order for each operator as a truncation point. The optimal enqueue order of the data structure is selected through final comparison.

[0012] Optionally, a dynamic programming method is used to determine the optimal operator enqueue order for each operator as a cutoff point, specifically:

[0013] Assume that computing resources, represented by ;

[0014] Assume that the data structure has operators, then the number of operators corresponding to each computing resource is expressed as , then: ;

[0015] Assume that an operator in the data structure is used as the truncation point, and the data structure stage is operators, then we have: ;

[0016] Assume that the computing resource type corresponding to each operator in the data structure is ,in, is the sequence number of the operator, is the serial number of the computing resource corresponding to the operator serial number;

[0017] Then the computing resource type corresponding to the operator at the truncation point is ;

[0018] Suppose the execution time of each operator in the data structure is ;

[0019] The elements of the DP array are as follows:

[0020]

[0021]

[0022]

[0023]

[0024] in, for A value between; for A value between; There are four elements in it, which respectively represent the time to execute to the current path, the time available for parallelization, the type of computing resources occupied, and whether the top-level operator is the next frame; It means that it does not occupy any resources; is the process of executing the optimal solution of the path according to the previous path; is the final optimal solution for the path, and is the optimal operator enqueue order under the current truncation point.

[0025] Optional computing resources, including but not limited to: NPU, DSP, GPU, CPU, FPGA.

[0026] The present invention also provides an image processing performance optimization system using an image processing performance optimization method, comprising:

[0027] Enqueue order determination module: used to build a unified data structure based on the data dependency between operators in each frame, and use dynamic programming to determine the enqueue order of operators in previous and next frames;

[0028] Operator distribution execution module: used to implement a scheduler based on the order of enqueuing. When the operator on which the head element of the execution queue in the scheduler depends is executed, the scheduler will dequeue the execution queue and distribute the dequeued operator to the execution thread of the corresponding computing resource for execution.

[0029] The present invention also provides an electronic device, comprising:

[0030] Memory for storing computer programs;

[0031] A processor is used to implement the steps of an image processing performance optimization method when executing a computer program.

[0032] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for optimizing image processing performance are implemented.

[0033] It can be seen from the above technical solutions that, compared with the prior art, the present invention proposes an image processing performance optimization method, system, device and storage medium. By adopting the parallel method of the front and back frames, the data congestion caused by the data dependency caused by frame-by-frame processing is reduced, and the utilization rate of heterogeneous computing resources is effectively improved, thereby improving the performance of image processing and the video frame rate of smart cameras, security equipment and the like. And the optimal queue order of the front and back frame operators is determined by the dynamic programming method, which further reduces the image processing time and improves the utilization rate of heterogeneous computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0036] Figure 2 This is a schematic diagram of the truncation of the previous and next frame operators in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Embodiment 1:

[0039] Embodiment 1 of the present invention discloses a method for optimizing image processing performance, such as Figure 1 As shown, including:

[0040] Step 1: Based on the data dependency between operators in each frame, a unified data structure is constructed, and the order in which operators of previous and next frames are queued is determined using dynamic programming.

[0041] The data structure of each frame is called OpInfo, and a specific id is assigned to the operator in each frame data structure, for example: xxxOpInfo, yyyOpInfo, etc.

[0042] The dynamic programming method is used to determine the order in which the operators of the previous and next frames enter the queue, specifically:

[0043] Each operator in the data structure is taken as a truncation point, and the dynamic programming method is used to determine the optimal operator enqueue order for each operator as a truncation point. The optimal enqueue order of the data structure is selected through final comparison.

[0044] The dynamic programming method is used to determine the optimal operator entry order for each operator as a cutoff point, specifically:

[0045] Assume that computing resources, represented by ;

[0046] Assume that the data structure has operators, then the number of operators corresponding to each computing resource is expressed as , then: ;

[0047] Assume that an operator in the data structure is used as the truncation point, and the data structure stage is operators, then we have: ;

[0048] Assume that the computing resource type corresponding to each operator in the data structure is ,in, is the sequence number of the operator, is the serial number of the computing resource corresponding to the operator serial number;

[0049] Then the computing resource type corresponding to the operator at the truncation point is ;

[0050] Suppose the execution time of each operator in the data structure is ;

[0051] The elements of the DP array are as follows:

[0052]

[0053]

[0054]

[0055]

[0056] in, for A value between; for A value between; There are four elements in it, which respectively represent the time to execute to the current path, the time available for parallelization, the type of computing resources occupied, and whether the top-level operator is the next frame; It means that it does not occupy any resources; The specific algorithm is a pseudo code for the process of executing the optimal solution of the path according to the previous path, as shown in Table 1. When min is used for comparison, it specifically compares the size of T.all_cost and takes the smaller one; is the final optimal solution for the path, and is the optimal operator enqueue order under the current truncation point.

[0057] Table 1 The specific algorithm is expressed in pseudo code

[0058] 1 # Pseudocode using Python syntax 2 # The four elements of T are 3 # 0 all_cost 4 # 1 now_cost 5 # 2 top_type 6 # 3 pingpong_flag 7 def Merge(T,t,R,pingpong_flag): 8 If T.pingpong == pingpong_flag: 9 T.all_cost += t 10 T.now_cost = t 11 T.top_type = R 12 else: 13 if T.top_type != R: 14 If T.now_cost>t: 15 T.now_cost -= t 16 else: 17 T.all_cost += (t - T.now_cost) 18 T.now_cost = t - T.now_cost 19 T.top_type = R 20 T.pingpong_flag = pingpong_flag 21 else: 22 T.all_cost += t 23 T.now_cost = t 24 T.pingpong_flag = pingpong_flag 25 return T

[0059] Step 2: Based on the order of enqueuing, implement a scheduler. When the operator that the head element of the execution queue in the scheduler depends on is executed, the scheduler will dequeue the execution queue and distribute the dequeued operator to the execution thread of the corresponding computing resource for execution.

[0060] Computing resources, including but not limited to: NPU, DSP, GPU, CPU, FPGA.

[0061] Embodiment 2:

[0062] Embodiment 2 of the present invention discloses a specific embodiment of using a dynamic programming method to determine the order in which operators of previous and next frames enter the queue, as follows:

[0063] like Figure 2 As shown, usually the same operator is executed on the previous and next frames. Assuming that the frame generation time of each frame is stable, the image processing process of each frame can be regarded as the running process of the previous and next frames after the single frame is truncated. The previous and next processes after truncation are regarded as two queues, and the so-called order specified is the order of dequeueing of the two queues. If 0 or 1 is used to represent the dequeueing of the two queues, then the dequeueing order can be expressed as a sequence of only 0 and 1 with a length of n nodes. Then according to the binomial theorem, the dequeueing order is kind.

[0064] Since there are too many sequences, it is impossible to directly brute force all solutions. The following two assumptions are made, that is, the execution order of operators does not affect the execution time of operators, and the execution time of operators is independent of each other. The latter operator must wait for the previous operator to complete execution, and there is no parallel execution in a single frame. Under these two prerequisites, the execution order of a part must correspond to a certain execution time. According to these two assumptions, for example, the four operators 0123, assuming that Figure 2 , taking 2 as the cutoff point, for the two execution orders 0213 and 0231 (only 01 is used to indicate that the dequeue order should be {1,0,1,0} and {1,0,0,1}), the optimal execution time of 02 is determined, so the number of traversals can be reduced by saving the intermediate results.

[0065] For a certain truncation point, assuming that the process is truncated into x operators and y operators, the optimal solution can be inferred through dynamic programming. Still taking 0123 (2 is the truncation point) as an example, assuming that 03 uses DSP and 12 uses NPU, then the DP array is shown in Table 2:

[0066] Table 2 DP array using 0123 (2 is the cutoff point) as an example

[0067]

[0068] As shown in Table 2, the DP array takes 0 as the starting point. Going right means dequeueing from queue 1, and going downward means dequeueing from queue 0. Each T not only represents the basic time required for execution, but also represents the current time and running type available for parallel operation, while the lowercase t represents the running time and type of a single operator. Each Merge is the process of calculating the current optimal solution based on the solutions on the left and above. The specific process is that when the two types are different, it is considered that they can be run in parallel, and the basic running time will be added to the parallel time of the two, and then the difference between the two times available for parallel operation is the new time available for parallel operation. The optimal value of a coordinate is to choose a better one from two possible paths. Specifically, assuming So ,right In terms of the two Merges, the two Merges are the same. Let's assume that the optimal path is from Come, then there is , and then use and To calculate , assuming ,So The optimal path comes from , and then assume The optimal path comes from , then the optimal path is , the four arrows represent 1010 respectively, so the optimal order is {1,0,1,0}. This method can be used to calculate the optimal execution time and execution path through the time of each operator, run it once in the debugging phase, and then pass the execution path as a parameter to the interface of the fixed execution order, so as to achieve the optimal execution efficiency of two frames in parallel.

[0069] Embodiment 3:

[0070] Embodiment 3 of the present invention discloses an image processing performance optimization system using an image processing performance optimization method, including:

[0071] Enqueue order determination module: used to build a unified data structure based on the data dependency between operators in each frame, and use dynamic programming to determine the enqueue order of operators in the previous and next frames;

[0072] Operator distribution execution module: used to implement a scheduler based on the order of enqueuing. When the operator on which the head element of the execution queue in the scheduler depends is executed, the scheduler will dequeue the execution queue and distribute the dequeued operator to the execution thread of the corresponding computing resource for execution.

[0073] Embodiment 4:

[0074] Embodiment 4 of the present invention discloses an electronic device, including:

[0075] Memory for storing computer programs;

[0076] A processor is used to implement the steps of an image processing performance optimization method when executing a computer program.

[0077] Embodiment 5:

[0078] Embodiment 5 of the present invention discloses that a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for optimizing image processing performance are implemented.

[0079] The embodiment of the present invention discloses an image processing performance optimization method, system, device and storage medium. By adopting the parallel method of the previous and next frames, the data congestion caused by the data dependency caused by frame-by-frame processing is reduced, and the utilization rate of heterogeneous computing resources is effectively improved, thereby improving the performance of image processing and the video frame rate of smart cameras, security equipment and the like. The optimal queue order of the previous and next frame operators is determined by the dynamic programming method, which further reduces the image processing time and improves the utilization rate of heterogeneous computing resources.

[0080] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0081] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for optimizing image processing performance, characterized in that: include: Step 1: Based on the data dependency between operators in each frame, a unified data structure is constructed, and the order in which operators of the previous and next frames are queued is determined using dynamic programming. Step 2: Based on the queue entry order, a scheduler is implemented. When the operator on which the head element of the execution queue in the scheduler depends is executed, the scheduler dequeues the execution queue and distributes the dequeued operator to the execution thread of the corresponding computing resource for execution; In step 1, the dynamic programming method is used to determine the order in which the operators of the previous and next frames enter the queue, specifically: Taking each operator in the data structure as a cutoff point, respectively, and using a dynamic programming method to determine the optimal operator enqueueing order when each operator is used as a cutoff point, and selecting the optimal enqueueing order of the data structure through final comparison and selection; The dynamic programming method is used to determine the optimal operator entry order for each operator as a cutoff point, specifically: Assume there are a kinds of computing resources, represented by R0, R1, R2, ..., R a-1 ; Assuming that there are b operators in the data structure, the number of operators corresponding to each computing resource is expressed as b0, b1, b2, ..., b a-1 , then: Assuming that a certain operator in the data structure is used as a truncation point, and the data structure is truncated into x+y operators, then: b=x+y; Assume that the computing resource type corresponding to the i-th operator in the data structure is Among them, i is the serial number of the operator, j i is the number of the computing resource corresponding to operator number i; Then the computing resource type corresponding to the operator at the truncation point is R jx+1 ; Suppose the execution time of the ith operator in the data structure is t i ; The elements of the DP array are as follows: T 00 ={0,0,R -1 ,1}; Where m is a value between 0 and y; n is a value between 0 and x; T mn There are four elements in R, which respectively represent the time to execute to the current path, the time available for parallelization, the type of computing resources occupied, and whether the top-level operator is the next frame; -1 Indicates that no resources are occupied; Merge is the process of executing the optimal solution of the path based on the previous path, then T yx is the final optimal solution for the path, and is the optimal operator enqueue order under the current truncation point.

2. The image processing performance optimization method according to claim 1, characterized in that: In step 2, the computing resources include: NPU, DSP, GPU, CPU, and FPGA.

3. An image processing performance optimization system using an image processing performance optimization method according to any one of claims 1 to 2, characterized in that: include: Enqueue order determination module: used to build a unified data structure based on the data dependency between operators in each frame, and use dynamic programming to determine the enqueue order of operators in previous and next frames; Operator distribution execution module: used to implement a scheduler based on the queue entry order. When the operator on which the head element of the execution queue in the scheduler depends is executed, the scheduler dequeues the execution queue and distributes the dequeued operator to the execution thread of the corresponding computing resource for execution.

4. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of an image processing performance optimization method as described in any one of claims 1-2 when executing the computer program.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing performance optimization method according to any one of claims 1 to 2 are implemented.

Images