An openvx computer vision acceleration method based on multi-dimension optimization

Abstract

This invention provides a multi-dimensional optimization method for accelerating OpenVX computer vision. It analyzes the time consumption of each operator node in the original OpenVX computation graph, obtaining the time consumption analysis result for each operator node. It also calculates the time consumption percentage of multiple operator nodes optimized by each optimization unit to determine the execution priority of each optimization unit. The original OpenVX computation graph is divided into multiple subgraphs according to the operator nodes optimized by each optimization unit. Each subgraph is assigned to a corresponding optimization unit for optimization, yielding the optimization output result of each optimization unit. Based on the pipeline sequence of the inference task represented in the original OpenVX computation graph, the optimization output result of the previous optimization unit is used as the input of the next optimization unit to complete the inference task. This invention optimizes the original OpenVX computation graph from different directions and dimensions, maximizing the overall performance improvement.

Description

Technical Field

[0001] This invention relates to the field of computer vision application technology, and more specifically, to a method for accelerating OpenVX computer vision based on multi-dimensional optimization. Background Technology

[0002] OpenVX is a cross-platform acceleration standard for computer vision applications, encompassing six core modules: a core module, a neural network (NN) module, a computer vision module, an image processing module, an object detection module, and a geometric transformation module. It is widely used in fields such as intelligent vision, autonomous driving, and industrial inspection. In practical applications, the various modules in OpenVX suffer from low computational efficiency, high memory overhead, frequent data movement, and insufficient parallelization. The core module, as the foundational layer, consumes 30-40% of the total processing time; the NN module consumes 20-30% in AI inference scenarios; the computer vision module, due to the high complexity of operations such as feature detection and optical flow calculation, consumes 15-25% of the time; and the remaining modules also suffer from limited overall performance due to their traditional algorithm implementation methods.

[0003] Existing OpenVX optimizations primarily focus on single-dimensional approaches, such as quantization for the neural network module, convolutional decomposition optimization for the image processing module, and simple multi-threaded scheduling for parallelization. These solutions only address performance issues in a single module or dimension, failing to provide a holistic, collaborative optimization across all OpenVX modules. Furthermore, existing solutions do not adequately integrate hardware characteristics into software optimization strategies, nor do they implement computation graph-level memory management and data reuse. This results in limited optimization effectiveness (typically performance improvements of no more than 20%), failing to meet the low-latency, high-throughput requirements of real-time computer vision applications.

[0004] Furthermore, existing OpenVX optimization technologies are mostly focused on low-level acceleration at the hardware driver layer, lacking in-depth reconstruction of the algorithm logic, data flow, and memory access patterns of each module. Moreover, a standardized multi-module collaborative optimization framework has not been formed, and the optimization strategies of different modules are independent of each other, making it impossible to maximize the overall performance improvement. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art by providing a multi-dimensional optimization method for accelerating OpenVX computer vision, thus overcoming the issues in the background technology.

[0006] This invention provides a multi-dimensional optimization method for accelerating OpenVX computer vision, comprising:

[0007] The time consumption of each operator node in the original OpenVX computation graph is analyzed to obtain the time consumption analysis results of each operator node. The time consumption ratio of multiple operator nodes optimized by each optimization unit is calculated to determine the execution priority of each optimization unit.

[0008] The original OpenVX computation graph is divided into multiple subgraphs according to the operator nodes optimized by the optimization unit. Each subgraph is assigned to the corresponding optimization unit for optimization, and the optimization output result of each optimization unit is obtained.

[0009] Based on the pipeline sequence of the inference task as represented in the original OpenVX computation graph, the optimization output of the previous optimization unit is used as the input of the next optimization unit to complete the inference task.

[0010] Based on the above technical solution, the present invention can also be improved as follows.

[0011] Optionally, the step of performing time consumption analysis on each operator node in the original OpenVX computation graph to obtain the time consumption analysis results for each operator node includes:

[0012] Instrument each operator node in the original OpenVX computation graph, add timestamps to the entry and exit points of each operator node, and collect the running metrics of each operator node, including execution time, input and output data size, memory access count, and hardware core utilization.

[0013] The average value of the results of multiple runs of each operator node is used to obtain the performance index of each operator node.

[0014] The method of calculating the time consumption percentage of multiple operator nodes optimized by each optimization unit to determine the execution priority of each optimization unit includes:

[0015] The operator nodes of the optimization unit are aggregated to obtain the set of operator nodes corresponding to each optimization unit;

[0016] Sum the running metrics of all operator nodes in the set of operator nodes corresponding to each optimization unit to obtain the time consumption of each optimization unit, and calculate the time consumption percentage of each optimization unit.

[0017] When the time consumption of an optimization unit exceeds a preset threshold, the operator node corresponding to the optimization unit will be the key optimization target, and the execution priority of each optimization unit will be determined based on the time consumption of each optimization unit.

[0018] Optionally, the optimization unit includes a core module optimization unit, a neural network module optimization unit, a computer vision module optimization unit, an image processing module optimization unit, an object detection module optimization unit, and a geometric transformation module optimization unit;

[0019] The core module optimization unit is used to perform graph kernel fusion, delayed memory allocation, and asynchronous parallel execution on the original OpenVX computation graph.

[0020] The neural network module optimization unit is used to perform INT8 quantization, convolution replacement, layer fusion, and sparse weight compression on the neural network model involved in the original OpenVX computation graph.

[0021] The computational vision module optimization unit is used to perform integral image pre-calculation, parallel feature detection, and hardware-accelerated optical flow operation on the input image.

[0022] The image processing module optimization unit is used to perform convolution separation and LUT-accelerated histogram equalization on the convolution operations involved in the input image.

[0023] The target detection module optimization unit is used to perform multi-scale parallel processing, cascaded early stopping operation and hardware NMS acceleration on the input image;

[0024] The geometric transformation module optimization unit is used to perform matrix pre-calculation, coordinate mapping table pre-calculation, and parallel image block processing for spatial position transformations involved in the inference task.

[0025] Optionally, the core module optimization unit is used to perform graph kernel fusion, delayed memory allocation, and asynchronous parallel execution on the original OpenVX computation graph, including:

[0026] Traverse the original OpenVX computation graph, identify consecutive basic operator nodes without data dependencies in the original OpenVX computation graph, and merge multiple consecutive basic operator nodes into a composite kernel.

[0027] Set the memory delay allocation properties for the input image and configure the asynchronous execution properties for the input image.

[0028] Optionally, the neural network module optimization unit is used to perform INT8 quantization, convolution replacement, layer fusion, and sparse weight compression on the neural network model involved in the original OpenVX computation graph, including:

[0029] Convert the 32-bit floating-point parameters of the neural network model to 8-bit fixed-point parameters to achieve INT8 quantization of the neural network model;

[0030] Replace the standard convolution in the neural network model with Winograd convolution;

[0031] Multiple consecutive minimal operators in a neural network model are fused into a single composite operator;

[0032] Pruning operations are performed on the weight matrix in the neural network model to achieve sparsity of the weight matrix.

[0033] Optionally, the computational vision module optimization unit is used to perform integral image pre-computation, parallel feature detection, and hardware-accelerated optical flow operations on the input image, including:

[0034] The integral image of the input image is pre-calculated;

[0035] The image is divided into blocks, and the image blocks are processed in parallel using OpenMP. Feature detection is performed on multiple image blocks in parallel.

[0036] Configure the optical flow node within the OpenVX framework and set its hardware acceleration properties.

[0037] Optionally, the image processing module optimization unit is used to perform convolution separation and LUT-accelerated histogram equalization on the convolution operations involved in the input image, including:

[0038] The 2D convolution involved in the image is decomposed into 1D horizontal convolution and 1D vertical convolution;

[0039] Pre-compute the lookup table to accelerate histogram equalization operations on images using LUTs.

[0040] Optionally, the target detection module optimization unit is used to perform multi-scale parallel processing, cascaded early stopping operations, and hardware NMS acceleration on the input image, including:

[0041] In object detection tasks, images of different scales are processed in parallel;

[0042] And during model training, negative samples are rejected early on:

[0043] Traverse each stage in the cascade detector;

[0044] Call the compute_stage_score function, pass in the feature values ​​of the current stage, and calculate the score of the current stage, stage_sum;

[0045] If the score stage_sum of the current stage is less than the threshold score stage.threshold of the current stage, then return false immediately and stop processing of subsequent stages.

[0046] Non-maximum suppression of images is performed in parallel using the GPU.

[0047] Optionally, the geometric transformation module optimization unit is used to perform matrix pre-computation, coordinate mapping table pre-computation, and image parallel block processing on the spatial position transformations involved in the inference task, including:

[0048] For spatial position transformations in reasoning tasks, the radial transformation matrix, perspective transformation matrix, and inverse matrix are pre-calculated, as well as the coordinate mapping table is pre-calculated;

[0049] And after dividing the image into blocks, perform geometric transformations on multiple image blocks in parallel.

[0050] Optionally, a global optimization scheduling unit is also included, which dynamically allocates computing resources to each optimization unit based on the time consumption ratio of each optimization unit and the hardware resource situation:

[0051] Priority is given to deploying high-time-consuming optimization units to high-performance hardware units to optimize the corresponding operator nodes. The high-time-consuming optimization units include core module optimization units and neural network module optimization units. The high-performance hardware units include NPU and GPU.

[0052] The low-time-consuming optimization unit is deployed on a multi-core CPU for parallel execution, and the low-time-consuming module includes a geometric transformation module optimization unit.

[0053] Establish a performance monitoring and feedback mechanism, perform task inference based on the multi-dimensional optimized OpenVX computation graph, compare the task inference performance with the target performance, and adjust the optimization parameters of each optimization unit in real time based on the comparison results.

[0054] This invention provides a multi-dimensional optimization method for accelerating OpenVX computer vision. It analyzes the time consumption of each operator node in the original OpenVX computation graph, obtaining the time consumption analysis result for each operator node. It also calculates the time consumption percentage of multiple operator nodes optimized by each optimization unit to determine the execution priority of each optimization unit. The original OpenVX computation graph is divided into multiple subgraphs according to the operator nodes optimized by each optimization unit. Each subgraph is assigned to a corresponding optimization unit for optimization, obtaining the optimization output result of each optimization unit. Based on the pipeline sequence of the inference task represented in the original OpenVX computation graph, the optimization output result of the previous optimization unit is used as the input of the next optimization unit to complete the inference task. This invention optimizes the original OpenVX computation graph from different directions and dimensions, maximizing the overall performance improvement. Attached Figure Description

[0055] Figure 1 A flowchart of a method for accelerating OpenVX computer vision based on multi-dimensional optimization is provided as an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram showing the connection of each optimization unit in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0058] As a standard for accelerating computer vision, OpenVX faces the following core technical challenges in practical deployments:

[0059] 1) Low computational efficiency: Each module adopts the traditional serial execution method. The core module performs multiple operations independently, resulting in frequent intermediate data transfer. The standard convolution multiplication and addition operations of the NN module are redundant. The feature calculation of the computer vision module is highly complex, resulting in a huge overall computational load.

[0060] 2) High memory overhead: Pre-allocated memory leads to severe fragmentation, intermediate results are repeatedly stored, model parameters are not compressed, and the overall memory usage is high (approximately 10GB before optimization).

[0061] 3) Insufficient parallelism: The parallel computing capabilities of multi-core CPUs, GPUs / NPUs and other hardware are not fully utilized, and data-level, task-level, and pipeline-level parallelism are not effectively combined;

[0062] 4) Poor hardware compatibility: The software optimization strategy is not customized for different hardware units (such as SIMD instruction sets, dedicated NPU, GPU optical flow units), and the hardware acceleration capabilities are not fully utilized;

[0063] 5) Low energy efficiency: Redundant computing and frequent memory access lead to excessive power consumption, which cannot meet the low power consumption requirements of embedded and edge computing scenarios.

[0064] Based on the aforementioned problems, this invention aims to design a multi-dimensional collaborative OpenVX optimization method. Through four optimization strategies—algorithm reconstruction, memory optimization, parallelization design, and hardware adaptation—it achieves a reduction of over 50% in processing time for each module, a reduction of over 60% in memory usage, and an overall performance improvement of over 60%, while ensuring that the algorithm accuracy is not significantly affected, thus meeting the performance and energy efficiency requirements of real-time computer vision applications.

[0065] like Figure 1 The diagram illustrates a flowchart of a multi-dimensional optimization method for accelerating OpenVX computer vision according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0066] Step 1: Perform time consumption analysis on each operator node in the original OpenVX computation graph, obtain the time consumption analysis results of each operator node, and count the time consumption ratio of multiple operator nodes optimized by each optimization unit to determine the execution priority of each optimization unit.

[0067] Understandably, the module time consumption analysis module performs a full-link performance analysis of the original OpenVX computation graph, identifying issues such as the execution time ratio of each operator / subgraph in the OpenVX computation graph, memory access bottlenecks, and insufficient parallelism.

[0068] This process involves instrumenting each operation node in the OpenVX computation graph and collecting data such as execution time, memory access count, and hardware utilization for each operation node. A time consumption heatmap is constructed, and statistics are collected by category for each optimization unit (core module optimization unit, NN module optimization unit, computer vision module optimization unit, image processing module optimization unit, object detection module optimization unit, and geometric transformation module optimization unit) to calculate the time consumption of each optimization unit. Based on the time consumption of each optimization unit, the optimization priority of all optimization units is sorted to guide the work focus of subsequent optimization units.

[0069] The analysis of the time consumption of operators / subgraphs in the original OpenVX computation graph includes the following steps:

[0070] (1) Data acquisition stage: Instrument the original OpenVX computation graph and add timestamps to the entry and exit of each node; collect the execution time, input and output data size, memory access times, hardware core utilization and other indicators of each node; take the average value of the results of multiple runs to eliminate the randomness of a single run.

[0071] (2) Bottleneck identification stage: Aggregate nodes according to each optimization unit to obtain the set of operator nodes corresponding to each optimization unit, calculate the total time consumption of the operator nodes corresponding to each optimization unit as the total time consumption of each optimization unit, and calculate the proportion of total time consumption of each optimization unit; identify modules whose time consumption proportion exceeds the threshold (e.g., 30%) as key optimization targets; further analyze the operators inside the key modules to identify specific performance bottlenecks (e.g., computationally intensive, memory-intensive, insufficient parallelism).

[0072] (3) Optimization order determination stage: Based on the severity of the bottleneck (time consumption ratio) and the optimization benefits (expected performance improvement), prioritize the optimization units of each module; prioritize the optimization of modules with high time consumption ratio and high optimization benefits (such as the NN module). For modules with dependencies (such as object detection depending on the output of image processing), the optimization order should take into account the dependency chain to avoid new bottlenecks after optimization.

[0073] (4) Output stage: Generate an optimization priority report to clarify the optimization focus and expected benefits of each module; pass the report to the global optimization and adjustment module as the basis for subsequent collaborative scheduling.

[0074] Step 2: Divide the original OpenVX computation graph into multiple subgraphs according to the operator nodes optimized by the optimization unit, assign each subgraph to the corresponding optimization unit for optimization, and obtain the optimization output result of each optimization unit.

[0075] Understandably, the performance bottlenecks of different modules vary significantly (e.g., neural networks are computationally intensive, while image processing is memory-intensive), and a single optimization strategy cannot address both simultaneously; domain-specific optimization is necessary. Therefore, targeted algorithm and implementation optimizations are performed for OpenVX operator subsets in different business scenarios.

[0076] Specifically, the original OpenVX computation graph can be divided into multiple subgraphs according to the operator nodes optimized by the optimization unit, and each subgraph can be assigned to the corresponding optimization unit for optimization.

[0077] Among them, see Figure 2 The optimization unit includes a core module optimization unit, a neural network module optimization unit, a computer vision module optimization unit, an image processing module optimization unit, an object detection module optimization unit, and a geometric transformation module optimization unit.

[0078] The core module optimization unit is used to perform graph kernel fusion, delayed memory allocation, and asynchronous parallel execution on the original OpenVX computation graph.

[0079] The neural network module optimization unit is used to perform INT8 quantization, convolution replacement, layer fusion, and sparse weight compression on the neural network model involved in the original OpenVX computation graph.

[0080] The computational vision module optimization unit is used to perform integral image pre-calculation, parallel feature detection, and hardware-accelerated optical flow operation on the input image.

[0081] The image processing module optimization unit is used to perform convolution separation and LUT-accelerated histogram equalization on the convolution operations involved in the input image.

[0082] The target detection module optimization unit is used to perform multi-scale parallel processing, cascaded early stopping operation and hardware NMS acceleration on the input image;

[0083] The geometric transformation module optimization unit is used to perform matrix pre-calculation, coordinate mapping table pre-calculation, and parallel image block processing for spatial position transformations involved in the inference task.

[0084] The optimization execution process for each optimization unit will be explained in detail below.

[0085] (1) Core module optimization unit (occupies 30-40% of the time, highest priority):

[0086] The core optimization strategy is graph kernel fusion + delayed memory allocation + asynchronous parallel execution, and the execution flow is as follows:

[0087] ① Computation graph analysis: Traverse the OpenVX computation graph to identify consecutive operation nodes without data dependencies (such as convolution → activation → pooling).

[0088] ② Kernel Fusion: Multiple consecutive operation nodes identified are merged into a composite kernel, reducing intermediate result storage and data movement. Code implementation:

[0089] A) The comments explain that the function of this code is to merge convolution, activation, and pooling operations into a single kernel;

[0090] B) Call the create_fused_kernel function, passing in the computation graph, input image, output image, and a list of kernel types including VX_KERNEL_CONVOLUTION (convolution), VX_KERNEL_ACTIVATION (activation), and VX_KERNEL_POOLING (pooling);

[0091] C) Assign the function's return value to a variable fused_node of type vx_node.

[0092] ③ Delayed memory allocation: Set the memory allocation attribute to avoid fragmentation caused by pre-allocation. The so-called "on-demand allocation" is a memory "lazy loading" mechanism implemented by the OpenVX framework in combination with AMD hardware features. Simply put, "memory is not allocated in advance, but only when it is actually needed, and not allocated when it is not needed."

[0093] Traditional pre-allocation: When an image object is created, the framework immediately allocates a full memory space for it (for example, a 1080P image requires about 6MB of memory), even if the image is not needed temporarily (for example, the computation graph has not yet reached this node).

[0094] Fragmentation: If a large amount of image memory that is not currently in use is pre-allocated at once, the memory space will be "occupied and not used". Subsequent operations that actually need memory may only be allocated to small, scattered memory blocks (fragmentation), or even trigger additional memory copies, reducing efficiency.

[0095] In this invention, the core of "on-demand allocation" of memory is "marking attributes + automatic triggering by the framework": the following A / B / C steps are just to turn on the delayed allocation switch, and the actual allocation is automatically completed by the framework when "the image is actually used".

[0096] The steps to set up on-demand memory allocation are as follows:

[0097] A) Call the vxSetImageAttribute function to set attributes for the image to be processed;

[0098] B) The image to be processed is set to the attribute VX_IMAGE_ATTRIBUTE_AMD_DEFERRED_ALLOCATION (AMD Delayed Allocation Attribute);

[0099] C) Set the delayed allocation attribute value to VX_TRUE, and the size of the value in bytes is sizeof(vx_bool) (the length of the vx_bool type in bytes).

[0100] Memory allocation timing for images to be processed: Allocate just enough memory only when the image is being computed, read, written, or mapped; do not allocate memory when not needed.

[0101] Core value: Avoid pre-allocated memory being "occupied" and reduce memory fragmentation, thereby improving memory utilization.

[0102] ④ Asynchronous parallel execution: The asynchronous execution attribute of the configuration graph schedules multi-threaded execution based on task priority.

[0103] A) Define a variable num_threads of type vx_uint32 and assign it the value 8;

[0104] B) Call the vxSetGraphAttribute function to set attributes for the computation graph;

[0105] C) Set the attribute to VX_GRAPH_ATTRIBUTE_AMD_ASYNC_EXECUTION (AMD asynchronous execution attribute);

[0106] D) The attribute value is the address of num_threads, and the size of the value in bytes is sizeof(vx_uint32) (the length of the vx_uint32 type in bytes).

[0107] (2) NN module optimization unit (occupies 20-30% of the time)

[0108] The core optimization strategy is INT8 quantization + Winograd convolution + layer fusion + sparse weight compression, and the process is as follows:

[0109] Quantization parameter configuration: Converts the 32-bit floating-point parameters of the neural network model to 8-bit fixed-point parameters, reducing computational and memory overhead.

[0110] vx_nn_quantization_params_t is a core structure in the OpenVX NN extension used to define quantization / dequantization parameters. It is specifically adapted to the numerical conversion rules for low-precision (such as INT8 / INT16) computations in neural network inference (based on the OpenVX NN standard specification).

[0111] ②Winograd convolution replacement: Replace the standard convolution in the neural network model with Winograd convolution to reduce the number of multiply-accumulate operations;

[0112] ③ Layer fusion: Fusion of Conv+BN+ReLU into a single operator, eliminating intermediate storage:

[0113] A) Define a variable fuse of type vx_nn_fuse_params_t and assign a value using aggregate initialization;

[0114] B) Set the member fuse_conv_bn_relu to VX_TRUE (enable the fusion of convolution, batch normalization, and ReLU activation);

[0115] C) Set the member fuse_add_relu to VX_TRUE (enable addition and ReLU-activated fusion);

[0116] D) Set the member fuse_depthwise_conv_bn to VX_TRUE (enable the fusion of depthwise convolution and batch normalization).

[0117] ④ Sparse weight compression: After pruning the model weights, they are stored in CSR format to reduce the amount of weight storage and computation.

[0118] Among them, "pruning model weights" is the core preliminary step of sparse weight compression. Essentially, it is to identify and remove weight parameters that "contribute very little" in the model, and only retain weights that have a significant impact on the model output, thereby achieving sparsity.

[0119] In model weights (such as the convolution kernel of a convolutional layer and the weight matrix of a fully connected layer), a large number of weights with values ​​close to 0 have almost no impact on the final prediction result. Pruning is to reset these "ineffective / inefficient" weights to 0 (or delete them directly), making the weight matrix a "sparse matrix" (most elements are 0), laying the foundation for sparse storage of CSR and reducing the amount of computation.

[0120] The core of weight pruning is to "first assess the importance of the weights, then remove the low-contribution weights, and finally fine-tune to restore accuracy." The core goal is to create a highly sparse weight matrix. In practice, the absolute value threshold method is preferred (it is simple and easy to implement). The pruning ratio needs to be adjusted according to the accuracy requirements to avoid over-pruning.

[0121] Pruning is a prerequisite for CSR format storage—only when pruning creates sufficient sparsity can CSR demonstrate its value in "reducing storage and computation".

[0122] (3) Optimization of computer vision module (15-25% of time):

[0123] The core optimization strategy is integral image pre-computation + parallel feature detection + hardware-accelerated optical flow:

[0124] ① Integral Image Pre-computation: For the image to be processed, the integral image is pre-computed. "Pre-computing the integral image" is a one-time preprocessing operation, performed only once before feature detection. Its core function is to generate a "pixel sum and auxiliary table." Its key value is reducing the computational complexity of the pixel sum in a rectangular region from O(n²) to O(1), significantly reducing the total time spent on feature detection. Integral image pre-computation reduces the computational complexity of rectangular features from O(n²) to O(1). The steps for pre-computing the integral image include:

[0125] A) Calculate the sum value of all element variables in the matrix feature (integral is a two-dimensional array), where y1: the row number of the upper boundary of the target rectangle, y2: the row number of the lower boundary of the target rectangle (y2≥y1), x1: the column number of the left boundary of the target rectangle, and x2: the column number of the right boundary of the target rectangle (x2≥x1). The formula for calculating the sum value is: sum=integral[y2][x2]-integral[y1-1][x2]-integral[y2][x1-1]+integral[y1-1][x1-1];

[0126] Specifically, the sum value is the element at position [y2][x2] in the integral 2D array, minus the element at position [y1-1][x2], minus the element at position [y2][x1-1], and finally add the element at position [y1-1][x1-1].

[0127] The advantages of pre-computing integral images are:

[0128] A) The core operation is to perform addition and subtraction operations on the four elements of the two-dimensional array integral to obtain the sum;

[0129] B) The calculation logic corresponds to the classic formula for summing regions in an integral graph, without needing to traverse all elements within the region.

[0130] ② Parallel Feature Detection: The image to be processed is divided into blocks to obtain image blocks. OpenMP is used to process the image blocks in parallel, and features of each image block are extracted to improve feature detection efficiency. The specific steps are as follows:

[0131] A) Use OpenMP directives to enable loop parallelization and specify dynamic scheduling to allocate loop tasks;

[0132] B) Using y as the loop variable, iterate from 0 to height with a step size of block_size;

[0133] C) In each loop, call the detect_features_in_block function, passing in the image, the current y-value, and the block_size, to detect features within the image block.

[0134] In short,

[0135] A) The core is to use OpenMP for dynamic parallel scheduling to traverse the image height direction in blocks according to block_size;

[0136] B) In each loop, the function is called to process the image patch feature detection corresponding to the starting position of y.

[0137] ③ Hardware-accelerated optical flow: Configure the hardware acceleration attributes of the optical flow node to utilize the GPU's dedicated optical flow unit:

[0138] A) Call the `vxOpticalFlowPyrLKNode` function, passing in the computation graph `graph`, the previous frame image `prev`, the current frame image `curr`, the previous frame feature points `prev_pts`, the current frame feature points `curr_pts`, and the parameter `params`, to create an optical flow computation node and assign it to the `vx_node` type variable `flow_node`. This step is the basic operation for "instantiating an optical flow computation task" in the OpenVX framework, generating an executable optical flow node.

[0139] B) Call the vxSetNodeAttribute function to set attributes for the flow_node node. This step is a prerequisite for enabling "hardware acceleration" for this optical flow computing node. The attribute configuration function reserves an entry point for custom configuration.

[0140] C) Set the attribute to VX_NODE_ATTRIBUTE_AMD_HW_ACCEL (AMD hardware acceleration attribute);

[0141] D) Set the attribute value to VX_TRUE, and the size of the value in bytes is sizeof(vx_bool) (the length of the vx_bool type in bytes).

[0142] The advantages of optical flow hardware acceleration are:

[0143] A) The core is to create an LK pyramid optical flow node and enable the AMD hardware acceleration attribute of that node;

[0144] B) First create an optical flow node, then enable hardware acceleration through property settings.

[0145] 4) Image processing module optimization (10-20% time commitment):

[0146] The core optimization strategy is to use separable convolution + LUT acceleration + structuring element decomposition:

[0147] ① Separate convolution: Decompose the 2D convolution into 1D horizontal + 1D vertical convolution to reduce computation. The steps for separating convolution are as follows:

[0148] A) Define a function separateable_convolve, with input (input image) and output (output image) of type vx_image as parameters, and no return value;

[0149] B) Call the create_temp_image function to create a temporary vx_image type image temp;

[0150] C) Call the convolve_horizontal function, passing in input and temp, to perform a horizontal convolution on the input image, and store the result in temp;

[0151] D) Call the convolve_vertical function, pass in temp and output, perform vertical convolution on temp, and store the result in output.

[0152] ②LUT Accelerated Histogram Equalization: Pre-compute the lookup table to avoid redundant calculations. The steps are as follows:

[0153] A) Define a uint8_t array lut with a length of 256. The results corresponding to all possible pixel values ​​from 0 to 255 are pre-calculated and stored in the lut array. When processing pixels, you can directly "look up the table" (lut [pixel value]). For 1 million pixels, only 1 million simple array value lookups are needed, which greatly reduces the amount of computation.

[0154] B) Using i as the loop variable, iterate from 0 to 255, corresponding to the grayscale values ​​of the image:

[0155] Calculate 255.0 multiplied by cdf[i] and add 0.5. Cast the value to uint8_t and assign it to lut[i]. Here, cdf[i] represents the value of the "cumulative distribution function" at i, ranging from 0 to 1 (the core parameter of histogram equalization, representing the proportion of pixels with gray values ​​≤ i to the total number of pixels).

[0156] C) Call the apply_lut function, passing in the image and the lookup table lut, and apply the lookup table to process the image.

[0157] (5) Optimization of the target detection module (10-20% time commitment):

[0158] The core optimization strategy is multi-scale parallelism + cascaded early stopping + hardware NMS acceleration:

[0159] ① Multi-scale pyramid parallel processing: Parallel processing of images at different scales improves detection efficiency.

[0160] A) Use the OpenMP directive #pragma omp parallel for to enable loop parallelization;

[0161] B) Use the image level as the loop variable and iterate from 0 to pyramid_levels (excluding).

[0162] C) In each loop, the process_pyramid_level function is called, passing in the pyramid data pyramid and the current level level, to process the data at that pyramid level.

[0163] ② Cascaded early stopping: Reject negative samples early to reduce invalid computation. The steps are as follows:

[0164] A) Traverse each stage in the cascade detector;

[0165] B) Call the compute_stage_score function, pass in the feature values ​​of the current stage (features), and calculate the score of the current stage (stage_sum);

[0166] C) If stage_sum is less than the threshold stage.threshold of the current stage, return false immediately (early stop mechanism to avoid executing the calculation of subsequent stages).

[0167] ③ Parallel NMS: Utilizes the GPU to perform non-maximum suppression in parallel, improving post-processing efficiency. The steps are as follows:

[0168] A) Define a boolean vector `suppress` with the same length as `detections`, and initialize all elements to `false`.

[0169] B) Enable loop parallelization using the OpenMP directive #pragma omp parallel for;

[0170] C) The outer loop iterates through all elements of detections, using i as the variable.

[0171] D) If suppress[i] is false, execute the inner loop: using j as the variable, iterate through the remaining elements of detections starting from i+1;

[0172] E) Calculate the intersection-union ratio (IoU) of detections[i] and detections[j]. If the result is greater than the threshold, set suppress[j] to true.

[0173] 6) Optimization of the geometric transformation module (5-15% time commitment):

[0174] The core optimization strategy is matrix pre-computation + lookup table remapping + parallel block processing:

[0175] Matrix pre-computation: Pre-compute the affine transformation matrix / perspective transformation matrix and its inverse to avoid real-time computation. The steps are as follows:

[0176] A) Call the cv::getRotationMatrix2D function, pass in the rotation center center, rotation angle angle, and scaling ratio, and generate a 2-row, 3-column floating-point affine transformation matrix affine_matrix (of type cv::Matx23f).

[0177] B) Calculate the inverse of the affine transformation matrix, assign the result to the inv_matrix variable of the same type, and comment that this operation is a cached inverse matrix.

[0178] ② Lookup table remapping: A coordinate mapping table is pre-calculated and reused directly in subsequent frames. The steps are as follows:

[0179] A) Declare two variables of type cv::Mat, map_x and map_y;

[0180] B) Call the cv::initUndistortRectifyMap function, passing in the camera matrix camera_matrix, distortion coefficients dist_coeffs, rotation matrix R, new camera matrix new_matrix, size, and data type CV_16SC2, to generate a correction mapping table and store it in map_x and map_y;

[0181] C) Call the cv::remap function to perform distortion correction on the input image using the specified interpolation method based on the generated map_x and map_y, and save the result to the output image.

[0182] ③ Parallel block processing: After dividing the image into blocks, geometric transformations are performed on multiple image blocks in parallel. The steps are as follows:

[0183] A) Define an integer constant TILE_SIZE with a value of 64;

[0184] B) Use the OpenMP directive #pragma omp parallel for collapse (2) to enable parallelization of the two loops;

[0185] C) The outer loop uses y as a variable, starting from 0 and traversing to height with a step size of TILE_SIZE;

[0186] D) The inner loop uses x as a variable, starting from 0 and traversing to width with a step size of TILE_SIZE;

[0187] E) In each loop, create a tile object of type cv::Rect based on x, y, and TILE_SIZE;

[0188] F) Call the transform_tile function, passing in input (tile), output (tile), and transform, to process the data in the tile region.

[0189] Step 3: According to the pipeline sequence of the inference task represented in the original OpenVX computation graph, the optimization output of the previous optimization unit is used as the input of the next optimization unit to complete the inference task.

[0190] Among them, such as Figure 2 As shown, after each optimization unit performs corresponding optimizations on the input image, the global optimization scheduling unit dynamically allocates computing resources based on the time consumption ratio of each optimization unit and the hardware resource situation. This mainly includes:

[0191] Priority is given to deploying high-time-consuming optimization units to high-performance hardware units to optimize the corresponding operator nodes. The high-time-consuming optimization units include core module optimization units and neural network module optimization units. The high-performance hardware units include NPU and GPU.

[0192] The low-time-consuming optimization unit is deployed on a multi-core CPU for parallel execution, and the low-time-consuming module includes a geometric transformation module optimization unit.

[0193] Based on the pipeline of the reasoning task, the input-output order of each optimization unit is determined, and the optimization output of the previous optimization unit is used as the input of the next optimization unit to complete the reasoning task.

[0194] Establish a performance monitoring and feedback mechanism, perform task inference based on the multi-dimensional optimized OpenVX computation graph, compare the task inference performance with the target performance, adjust the optimization parameters of each optimization unit in real time based on the comparison results, and use each optimization unit to iteratively optimize the OpenVX computation graph until the optimized performance indicators reach the target performance.

[0195] This invention provides a multi-dimensional optimization method for accelerating OpenVX computer vision, which has the following beneficial effects:

[0196] 1) Significant global optimization effect: Layered optimization is performed on the time consumption ratio of multiple optimization units in OpenVX, prioritizing the resolution of performance bottlenecks of high-time-consuming optimization units, while also taking into account the collaboration between multiple optimization units. Compared with a single-dimensional optimization scheme, the overall processing time is reduced by more than 50%, and the performance is improved by more than 60%, which can meet the low latency requirements of real-time computer vision applications.

[0197] 2) Significantly improved memory efficiency: Through strategies such as delayed allocation, memory pools, and data reuse, memory usage is reduced by more than 6GB, memory fragmentation is resolved, cache hit rate is improved, and data access latency is further reduced;

[0198] 3) Strong hardware adaptability: The software and hardware co-optimization strategy can adapt to different hardware units such as multi-core CPU, GPU / NPU, and DSP, fully releasing the hardware acceleration capabilities. Compared with pure software optimization solutions, the energy efficiency ratio is improved by more than 2.5 times, which is suitable for low-power scenarios such as embedded systems and edge computing.

[0199] 4) Balance between accuracy and performance: When using approximate calculation strategies such as 8-bit quantization and cascaded early stopping, the algorithm accuracy loss is kept within an acceptable range (usually accuracy decrease <5%) by means of layer-by-layer quantization and high-precision preservation of sensitive layers, thus balancing performance and business needs.

[0200] 5) High scalability: The modular optimization framework can flexibly add new optimization strategies or adapt to new hardware units without reconstructing the overall architecture, making it suitable for different versions of the OpenVX standard and diverse computer vision application scenarios.

[0201] 6) Significantly improved throughput: The hierarchical parallel design increases throughput by more than 3 times, enabling the processing of higher resolution and higher frame rate image data, meeting the needs of high-definition real-time visual processing.

[0202] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0203] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0205] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0206] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0207] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0208] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for accelerating OpenVX computer vision based on multi-dimensional optimization, characterized in that, include: The time consumption of each operator node in the original OpenVX computation graph is analyzed to obtain the time consumption analysis results of each operator node. The time consumption ratio of multiple operator nodes optimized by each optimization unit is calculated to determine the execution priority of each optimization unit. The original OpenVX computation graph is divided into multiple subgraphs according to the operator nodes optimized by the optimization unit. Each subgraph is assigned to the corresponding optimization unit for optimization, and the optimization output result of each optimization unit is obtained. According to the pipeline sequence of the inference task represented in the original OpenVX computation graph, the optimization output of the previous optimization unit is used as the input of the next optimization unit to complete the inference task, which is the object detection task.

2. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 1, characterized in that, The time consumption analysis of each operator node in the original OpenVX computation graph is performed to obtain the time consumption analysis results for each operator node, including: Instrument each operator node in the original OpenVX computation graph, add timestamps to the entry and exit points of each operator node, and collect the running metrics of each operator node, including execution time, input and output data size, memory access count, and hardware core utilization. The average value of the results of multiple runs of each operator node is used to obtain the performance index of each operator node. The method of calculating the time consumption percentage of multiple operator nodes optimized by each optimization unit to determine the execution priority of each optimization unit includes: The operator nodes of the optimization unit are aggregated to obtain the set of operator nodes corresponding to each optimization unit; Sum the running metrics of all operator nodes in the set of operator nodes corresponding to each optimization unit to obtain the time consumption of each optimization unit, and calculate the time consumption percentage of each optimization unit. When the time consumption of an optimization unit exceeds a preset threshold, the operator node corresponding to the optimization unit will be the key optimization target, and the execution priority of each optimization unit will be determined based on the time consumption of each optimization unit.

3. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 1, characterized in that, The optimization unit includes a core module optimization unit, a neural network module optimization unit, a computer vision module optimization unit, an image processing module optimization unit, an object detection module optimization unit, and a geometric transformation module optimization unit; The core module optimization unit is used to perform graph kernel fusion, delayed memory allocation, and asynchronous parallel execution on the original OpenVX computation graph. The neural network module optimization unit is used to perform INT8 quantization, convolution replacement, layer fusion, and sparse weight compression on the neural network model involved in the original OpenVX computation graph. The computational vision module optimization unit is used to perform integral image pre-calculation, parallel feature detection, and hardware-accelerated optical flow operation on the input image. The image processing module optimization unit is used to perform convolution separation and LUT-accelerated histogram equalization on the convolution operations involved in the input image. The target detection module optimization unit is used to perform multi-scale parallel processing, cascaded early stopping operation and hardware NMS acceleration on the input image; The geometric transformation module optimization unit is used to perform matrix pre-calculation, coordinate mapping table pre-calculation, and parallel image block processing for spatial position transformations involved in the inference task.

4. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, The core module optimization unit is used to perform graph kernel fusion, delayed memory allocation, and asynchronous parallel execution on the original OpenVX computation graph, including: Traverse the original OpenVX computation graph, identify consecutive basic operator nodes without data dependencies in the original OpenVX computation graph, and merge multiple consecutive basic operator nodes into a composite kernel. Set the memory delay allocation properties for the input image and configure the asynchronous execution properties for the input image.

5. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, The neural network module optimization unit is used to perform INT8 quantization, convolution replacement, layer fusion, and sparse weight compression on the neural network model involved in the original OpenVX computation graph, including: Convert the 32-bit floating-point parameters of the neural network model to 8-bit fixed-point parameters to achieve INT8 quantization of the neural network model; Replace the standard convolution in the neural network model with Winograd convolution; Multiple consecutive minimal operators in a neural network model are fused into a single composite operator; Pruning operations are performed on the weight matrix in the neural network model to achieve sparsity of the weight matrix.

6. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, The computational vision module optimization unit is used to perform integral image pre-computation, parallel feature detection, and hardware-accelerated optical flow operations on the input image, including: The integral image of the input image is pre-calculated; The image is divided into blocks, and the image blocks are processed in parallel using OpenMP. Feature detection is performed on multiple image blocks in parallel. Configure the optical flow node within the OpenVX framework and set its hardware acceleration properties.

7. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, The image processing module optimization unit is used to perform convolution separation and LUT-accelerated histogram equalization on the convolution operations involved in the input image, including: The 2D convolution involved in the image is decomposed into 1D horizontal convolution and 1D vertical convolution; Pre-compute the lookup table to accelerate histogram equalization operations on images using LUTs.

8. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, The target detection module optimization unit is used to perform multi-scale parallel processing, cascaded early stopping operations, and hardware NMS acceleration on the input image, including: In object detection tasks, images of different scales are processed in parallel; And during model training, negative samples are rejected early on: Traverse each stage in the cascade detector; Call the compute_stage_score function, pass in the feature values ​​of the current stage, and calculate the score of the current stage, stage_sum; If the score stage_sum of the current stage is less than the threshold score stage.threshold of the current stage, then return false immediately and stop processing of subsequent stages. Non-maximum suppression of images is performed in parallel using the GPU.

9. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, The geometric transformation module optimization unit is used to perform matrix pre-computation, coordinate mapping table pre-computation, and parallel image block processing on spatial position transformations involved in the inference task, including: For spatial position transformations in reasoning tasks, the radial transformation matrix, perspective transformation matrix, and inverse matrix are pre-calculated, as well as the coordinate mapping table is pre-calculated; And after dividing the image into blocks, perform geometric transformations on multiple image blocks in parallel.

10. The OpenVX computer vision acceleration method based on multi-dimensional optimization according to claim 3, characterized in that, It also includes a global optimization scheduling unit, which dynamically allocates computing resources to each optimization unit based on the time consumption ratio of each optimization unit and the hardware resource situation: Priority is given to deploying high-time-consuming optimization units to high-performance hardware units to optimize the corresponding operator nodes. The high-time-consuming optimization units include core module optimization units and neural network module optimization units. The high-performance hardware units include NPU and GPU. The low-time-consuming optimization unit is deployed on a multi-core CPU for parallel execution, and the low-time-consuming module includes a geometric transformation module optimization unit. Establish a performance monitoring and feedback mechanism, perform task inference based on the multi-dimensional optimized OpenVX computation graph, compare the task inference performance with the target performance, and adjust the optimization parameters of each optimization unit in real time based on the comparison results.

Images