A software-hardware collaborative optimization method for video structured analysis system based on parameter value selection

Abstract

The present invention belongs to the field of video analysis technology, and discloses a method for software-hardware collaborative optimization of a video structured analysis system based on parameter value selection, including step 1: construction of multi-module algorithm control parameters and implementation parameter sets; based on the industry's mainstream YOLO+byteTrack+ResNet+SIFT CDVS feature video structured analysis system framework, combined with the encoding algorithm characteristics and pipeline constraints of heterogeneous multi-computing platforms, a hardware-friendly algorithm framework is selected for each module; step 2: multi-module coupling parameter optimization value selection; according to the pipeline throughput constraint, the control parameters of a certain concurrency granularity and concurrency intensity are configured to achieve a balance between system throughput and hardware consumption. The present invention supports the joint optimization of six algorithm modules, supports the joint optimization of target performance such as bit rate, computational complexity, algorithm accuracy, and system processing retrieval delay; and realizes the joint optimization of multi-target performance of multiple algorithm modules.

Description

Technical Field

[0001] The present invention belongs to the technical field of video analysis, and in particular relates to a software-hardware collaborative optimization method for a video structured analysis system based on parameter value selection. Background Art

[0002] In recent years, big data applications such as smart transportation and security monitoring have experienced rapid development. Artificial intelligence algorithms and high-performance computing resources have significantly promoted the implementation of intelligent applications. The video structured system based on deep learning, big data, and cloud storage is positioned as a "massive monitoring filter, capturing valuable information." As a means of upgrading video surveillance systems, it can provide users with video, image, and feature analysis functions and services, extracting valuable information from massive surveillance image resources, filtering redundant data, and forming a structured video data accumulation to support intelligent businesses such as smart cities and intelligent transportation. It implements intelligent video analysis and data mining, transforming videos from manual sampling to intelligent extraction, and realizing intelligent information extraction, analysis, and application. This frees people from the labor of watching video surveillance and allows them to carry out efficient and accurate practical applications of video images.

[0003] The video structured analysis system utilizes video structured description technology and intelligent image analysis technology to analyze and process real-time or offline video recordings. It connects to the video image information resource exchange and sharing platform based on the GB / T28181 standard to obtain video streams. The video structured analysis system extracts semantic objects such as people, vehicles, and moving objects from massive video streams, transforming massive amounts of unstructured video and semi-structured images into valuable compact unstructured and structured feature data. This compact feature information and structured data are then connected to the Public Security Video Image Information Sharing Platform of the Comprehensive Administration Office via WebService or Kafka interfaces. The video structured application system provides basic, specialized, and system management functions for video structured applications, including fuzzy search, conditional extraction, person-based search, and trajectory playback.

[0004] The amount of video data is extremely large. The number of videos that need to be analyzed for city-level video collaborative perception applications may reach millions. Video structured analysis calculations include video decoding, target detection, tracking, image optimization, feature extraction, recognition, retrieval, etc. When designing a video structured analysis system, it is necessary to consider multi-objective performance, including equipment cost, power consumption, storage and transmission cost, as well as system processing retrieval delay and algorithm accuracy (such as retrieval accuracy). Multi-objective performance optimization is a constraint that needs to be considered in video structured analysis systems, such as Figure 1 shown.

[0005] As mentioned above, the video structured analysis system contains multiple customizable modules, and its algorithms allow users to customize and optimize them according to their needs. Algorithm optimization is very important for the performance of the entire video analysis system. In the video analysis system, there are complex relationships between modules such as target detection, target tracking, image optimization, and feature extraction. Algorithm customization and optimization need to jointly consider the interaction mechanism between modules. When optimizing algorithms for heterogeneous hardware platforms, it is necessary to consider the latency and accuracy constraints of edge-cloud analysis retrieval. Under the premise of meeting the constraints, it is necessary to evaluate the target performance of different computing performances, such as device cost, power consumption cost, storage cost, and transmission cost. A comprehensive balance should be made between target performance such as rate-distortion performance, algorithm accuracy, hardware (computing, storage) resource consumption, throughput, and external memory bandwidth. It can be seen that the design and optimization of video structured analysis algorithms is a problem of joint optimization of multiple customizable module algorithms and system hardware and software collaboration.

[0006] Algorithm optimization for heterogeneous computing platforms involves two levels of tasks: algorithm flow optimization and algorithm implementation optimization. The former, similar to software-implemented video analysis systems, affects rate-distortion-algorithm accuracy performance; the latter only affects algorithm implementation complexity and offers significant room for optimization. Video structured analysis systems can customize module algorithm content optimization. This requires determining the algorithm control flow and framework based on the characteristics of each module's algorithm, and rationally configuring the module algorithm control parameters (algorithm content parameters) to achieve rate-distortion-accuracy constraint optimization. Algorithm implementation optimization primarily involves balancing system throughput and hardware consumption by configuring control parameters (algorithm implementation parameters) for a specific concurrency granularity and concurrency intensity, based on pipeline throughput constraints.

[0007] Existing methods mainly target multiple customizable algorithm modules for video coding, and mainly consider the rate-distortion-complexity performance optimization of video coding compression. However, there is currently a lack of a multi-module and multi-objective performance collaborative optimization method for video analysis systems that comprehensively considers multiple algorithm modules such as target detection, target tracking, image optimization, and feature calculation, and comprehensively considers target performance such as preparation cost, power consumption cost, storage cost, transmission cost, system processing retrieval delay, and algorithm accuracy, and considers the adaptation of heterogeneous target computing platforms to software and hardware collaboration. Summary of the Invention

[0008] The purpose of the present invention is to provide a software-hard collaborative optimization method for a video structured analysis system based on parameter value selection to solve the above-mentioned technical problems.

[0009] To solve the above technical problems, the specific technical solution of the software-hardware collaborative optimization method of the video structured analysis system based on parameter value selection of the present invention is as follows:

[0010] A method for software-hardware collaborative optimization of a video structured analysis system based on parameter value selection includes the following steps:

[0011] Step 1: Construction of multi-module algorithm control parameters and implementation parameter sets;

[0012] The video structured analysis system includes customizable algorithm modules: target detection, target tracking, image optimization, structured analysis, deep feature extraction, and target image encoding. Based on the industry's mainstream YOLO+byteTrack+ResNet+SIFT CDVS feature video structured analysis system framework, combined with the encoding algorithm characteristics and pipeline constraints of heterogeneous multi-computing platforms, a hardware-friendly algorithm framework is selected for each module.

[0013] Step 2: Optimization and selection of multi-module coupling parameters;

[0014] Based on the pipeline throughput constraints, a balance between system throughput and hardware consumption is achieved by configuring control parameters for a certain concurrency granularity and concurrency intensity.

[0015] Furthermore, the algorithm control parameters in step 1 include algorithm process parameters that affect rate-distortion-precision performance, and algorithm implementation parameters that only affect resource consumption performance, which include concurrency granularity and intensity related to module implementation.

[0016] Furthermore, step 1 includes pipeline algorithm optimization: first, it is necessary to determine which algorithm parameters to be selected. For each algorithm parameter, it is necessary to determine the discretized candidate values. This process is called discretization of algorithm parameter values. The principle of discretization of parameter values is: there should be a certain degree of distinction between candidate discrete values, but the distinction should not be too large; too much precision will lead to an increase in discrete values, resulting in an excessively large dimension of the search space, and a compromise needs to be made between distinction and complexity.

[0017] Furthermore, in step 1, all algorithm control parameters are divided into 6 modules, namely target detection, target tracking, image optimization, structured analysis, deep feature extraction, and target image encoding; these modules are marked as X i (i=1~6), module X i There is M i Algorithm parameters to be selected, for module X i The jth parameter p in ij , (j=1~M i ), is the parameter p ij Determine N ij The discrete values are represented as v(i,j,k), (k=1~N ij ), the pipeline algorithm optimization is based on the multi-objective performance constraints of rate-distortion-resource consumption-processing delay, and all parameters p ijSelect one value respectively to form a suitable parameter value combination. The parameter combination selection is a multi-module, multi-algorithm parameter, and multi-objective performance combination constraint problem.

[0018] Furthermore, the step 2 combines the target performance parameter homogeneity measurement method of equipment cost, power consumption cost, storage cost, and transmission cost to estimate the bit rate-computational complexity cost of the video structured analysis system.

[0019] Furthermore, step 2 includes:

[0020] The cost of bit rate, computational complexity, and algorithm accuracy loss of the video structured analysis system is as follows:

[0021]

[0022] The cost J takes into account the bit rate, computational complexity, and algorithm accuracy; another constraint is the system processing retrieval delay τ;

[0023] The optimization task of the video structured algorithm with multi-objective performance constraint optimization is equivalent to satisfying a specific processing delay τ tar Under the constraints, find a suitable algorithm parameter combination The optimal parameter combination Minimize the cost J of system bit rate-computational complexity-algorithm accuracy loss:

[0024]

[0025] Furthermore, step 2 includes optimizing the parameter p m_init Select the algorithm as follows:

[0026] Algorithm 1: Optimize parameter p m_init Select Algorithm

[0027] enter: Output: Initial parameter combination: Parameters: System processing retrieval delay: τ tar , rate-distortion cost tolerance: e tar , the maximum number of cycles: T loop

[0028] Multiple module correlation coupling coefficients θ i

[0029]

[0030]

[0031] There is a certain coupling relationship between multiple modules, θ i It represents the degree of coupling between the current module and other modules.i The values are ordered from small to large to determine the decision priority of each module algorithm.

[0032] Furthermore, assuming that the initial parameter selection algorithm sets N target compression rate ranges

[0033] (n=1,2,…,N), Algorithm 1 gives the initial parameter optimization selection algorithm process, where, The rate-distortion cost is expressed as like The mth parameter in p m becomes The corresponding bit rate-computational complexity-algorithm accuracy cost is expressed as For each non-discrete parameter, it is discretized into K m is the number of candidate solutions for the mth parameter. By using a set of test sequences to calculate the integral area of the rate-computational complexity-algorithm accuracy cost curve within a certain rate range, the rate-distortion cost of all possible parameter combination solutions is estimated offline. Then, an iterative algorithm is used for parameter selection to converge to the predefined rate-computational complexity-algorithm accuracy cost.

[0034] The software-hardware collaborative optimization method for a video structured analysis system based on parameter value selection of the present invention has the following advantages:

[0035] 1. This paper proposes a multi-module, multi-objective performance algorithm optimization method that supports device-edge-cloud collaborative video structured analysis. It supports the joint optimization of six algorithm modules: target detection, target tracking, image optimization, structured analysis, deep feature extraction, and target image encoding. It also supports the joint optimization of target performance such as bit rate (storage cost, transmission cost), computational complexity (equipment cost, power consumption cost), algorithm accuracy, and system processing and retrieval delay.

[0036] 2. Support joint optimization of algorithm control parameters and algorithm implementation parameters, and realize joint optimization of multi-objective performance of multiple algorithm modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram for multi-objective performance constraint optimization;

[0038] Figure 2 This is a schematic diagram of a customizable module for multi-algorithm video structured analysis of the present invention;

[0039] Figure 3 The multi-module algorithm process parameters and algorithm implementation parameters of the present invention Schematic diagram;

[0040] Figure 4This is a schematic diagram of the end-edge-cloud converged storage for video structured analysis of the present invention;

[0041] Figure 5 Schematic diagram of the bit rate (transmission and storage)-computational complexity model of the video structured analysis system of the present invention. DETAILED DESCRIPTION

[0042] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of the software and hardware collaborative optimization method of the video structured analysis system based on parameter value selection of the present invention in conjunction with the accompanying drawings.

[0043] like Figure 2 As shown, the video structured analysis system described in the present invention includes customizable algorithm modules such as target detection, target tracking, image optimization, structured analysis, deep feature extraction, target image encoding, etc.

[0044] The video structured analysis system allows for customized module algorithm content optimization. This requires determining the algorithm control flow and framework based on the module algorithm's characteristics. This requires proper configuration of the module algorithm control parameters (algorithm flow parameters) to achieve rate-distortion-algorithm accuracy constraint optimization. Furthermore, algorithm implementation optimization primarily involves balancing system throughput and hardware consumption by configuring control parameters (algorithm implementation parameters) at a specific concurrency granularity and concurrency intensity, based on pipeline throughput constraints.

[0045] This invention is based on the industry's mainstream YOLO+byteTrack+ResNet+SIFT CDVS feature video structured analysis system framework, combined with the encoding algorithm characteristics and pipeline constraints of heterogeneous multi-computing platforms, to select a hardware-friendly algorithm framework for each module. After determining the basic algorithm framework and pipeline structure, the specific algorithm details of each module still need to be further customized and optimized. This invention transforms the algorithm optimization problem of multi-module association and multi-objective performance constraints into a series of algorithm parameter value selection problems under rate-distortion-resource consumption constraints. Specifically, it includes the following steps:

[0046] Step 1: Multi-module algorithm control parameters and implementation parameter set construction:

[0047] Algorithm parameters include algorithm flow parameters that affect rate-distortion-precision performance, as well as algorithm implementation parameters that solely affect resource consumption. These parameters include the concurrency granularity and intensity associated with module implementation. Determining the algorithm parameters and building a pipeline algorithm verification model with configurable parameters is the basis for selecting algorithm parameter values.

[0048] Pipeline algorithm optimization first requires determining the candidate algorithm parameters. For each algorithm parameter, candidate discretized values must be determined. This process is called discretization of the algorithm parameter values. The principle of discretization of candidate parameter values in this invention is to ensure that the candidate discrete values have a certain degree of discrimination, while also avoiding excessive discrimination. Excessive discrimination leads to an increase in the number of discrete values, which in turn increases the dimensionality of the search space. A compromise must be struck between discrimination and complexity.

[0049] like Figure 3 As shown in Figure 1, to reduce the search dimension, all algorithm parameters are divided into 6 modules, namely target detection, target tracking, image optimization, structured analysis, deep feature extraction, and target image encoding. These modules are marked as X i (i=1~6), module X i There is M i Algorithm parameters to be selected, for module X i The jth parameter p in ij , (j=1~M i ), is the parameter p ij Determine N ij The discrete values are represented as v(i,j,k), (k=1~N ij ). The pipeline algorithm optimization is to optimize all parameters p under the multi-objective performance constraints of rate-distortion-resource consumption-processing delay. ij Select a value for each parameter to form a suitable parameter value combination. Parameter combination selection is a multi-module, multi-algorithm parameter, and multi-objective performance constraint problem.

[0050] Step 2: Optimization and selection of multi-module coupling parameters

[0051] like Figure 4 As shown, referring to the applicant's published patent "A multi-objective performance evaluation method for end-edge-cloud collaborative video group intelligence perception", a method combining the computing, storage, transmission, and power consumption of each video end-edge-cloud structured analysis processing is proposed, which combines the target performance parameters such as equipment cost, power consumption cost, storage cost, and transmission cost to estimate the bit rate (transmission and storage) - computational complexity cost of the video structured analysis system, as shown in the figure. Figure 5 shown.

[0052] The cost of video structured analysis system bit rate (transmission and storage) - computational complexity - algorithm accuracy loss is as follows:

[0053]

[0054] The cost J takes into account the bit rate (storage cost, transmission cost), computational complexity (equipment cost, power consumption cost) and algorithm accuracy; another constraint is the system processing retrieval delay τ.

[0055] The optimization task of the video structured algorithm with multi-objective performance constraint optimization is equivalent to satisfying a specific processing delay τ tar Under the constraints, find a suitable algorithm parameter combination (Optimal parameter combination ), so that the system bit rate - computational complexity - algorithm accuracy loss cost J is minimized, that is:

[0056]

[0057] Algorithm 1: Optimize parameter p m_init Select Algorithm

[0058] enter:

[0059] Output: Initial parameter combination:

[0060] Parameters: System processing retrieval delay: τ tar , rate-distortion cost tolerance: e tar , the maximum number of cycles: T loop

[0061] Multiple module correlation coupling coefficients θ i

[0062]

[0063]

[0064] There is a certain coupling relationship between multiple modules, θ i It represents the degree of coupling between the current module and other modules. i The values are ordered from small to large to determine the decision priority of each module algorithm.

[0065] Assume that the initial parameter selection algorithm sets N target compression rate ranges Algorithm 1 gives the initial parameter optimization selection algorithm process of the present invention, where: The rate-distortion cost is expressed as like The mth parameter in p m becomes The corresponding bit rate-computational complexity-algorithm accuracy cost is expressed as For each non-discrete parameter, it is discretized into K mis the number of candidate solutions for the mth parameter. By using a set of test sequences to calculate the integral area of the rate-computational complexity-algorithm accuracy cost curve within a certain rate range, the rate-distortion cost of all possible parameter combinations is estimated offline. An iterative algorithm is then used to select parameters to converge to the predefined rate-computational complexity-algorithm accuracy cost.

[0066] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection, characterized in that: The steps include: Step 1: Construction of multi-module algorithm control parameters and implementation parameter sets; The video structured analysis system includes customizable algorithm modules: target detection, target tracking, image optimization, structured analysis, deep feature extraction, and target image encoding. Based on the industry's mainstream YOLO+byteTrack+ResNet+SIFT CDVS feature video structured analysis system framework, combined with the encoding algorithm characteristics and pipeline constraints of heterogeneous multi-computing platforms, a hardware-friendly algorithm framework is selected for each module. Step 2: Optimization and selection of multi-module coupling parameters; Based on pipeline throughput constraints, a balance between system throughput and hardware consumption is achieved by configuring control parameters for a certain concurrency granularity and concurrency intensity. The cost of bit rate, computational complexity, and algorithm accuracy loss of the video structured analysis system is as follows: Among them, RC cost is the rate-complexity cost; λ is the Lagrange multiplier coefficient; L(D) is the loss of algorithm accuracy under distortion D; It is the total system bit rate-computational complexity-algorithm accuracy loss cost under the condition of parameter p; The cost J takes into account the bit rate, computational complexity, and algorithm accuracy; another constraint is the system processing retrieval delay τ; The optimization task of the video structured algorithm with multi-objective performance constraint optimization is equivalent to satisfying a specific processing delay τ tar Under the constraints, find a suitable algorithm parameter combination The optimal parameter combination Minimize the cost J of system bit rate-computational complexity-algorithm accuracy loss: in, is the minimum cost of system bit rate-computational complexity-algorithm accuracy loss, τ is the system processing retrieval delay; τ tar is a specific processing delay, represents the optimal parameter combination.

2. The method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection according to claim 1 is characterized in that: The algorithm control parameters in step 1 include algorithm process parameters that affect rate-distortion-precision performance, and algorithm implementation parameters that only affect resource consumption performance, which include concurrency granularity and intensity related to module implementation.

3. The method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection according to claim 1, characterized in that: The step 1 includes pipeline algorithm optimization: first, the algorithm parameters to be selected must be determined. For each algorithm parameter, the discretized candidate values must be determined. This process is called discretization of algorithm parameter values. The principle of discretization of parameter values is: there must be a certain degree of distinction between candidate discrete values, but excessive distinction must be avoided; being too precise will increase the number of discrete values, resulting in an excessively large search space dimension, and a compromise must be made between distinction and complexity.

4. The method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection according to claim 1, characterized in that: In step 1, all algorithm control parameters are divided into 6 modules, namely target detection, target tracking, image optimization, structured analysis, deep feature extraction, and target image encoding; These modules are marked with an X i (i=1~6), module X i There is M i Algorithm parameters to be selected, for module X i The jth parameter p in ij , (j=1~M i ), is the parameter p ij Determine N ij The discrete values are represented as v(i,j,k), (k=1~N ij ), the pipeline algorithm optimization is based on the multi-objective performance constraints of rate-distortion-resource consumption-processing delay, and all parameters p ij Select one value respectively to form a suitable parameter value combination. The parameter combination selection is a multi-module, multi-algorithm parameter, and multi-objective performance combination constraint problem.

5. The method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection according to claim 1, characterized in that: The step 2 combines the target performance parameter homogeneity measurement method of equipment cost, power consumption cost, storage cost, and transmission cost to estimate the bit rate-computational complexity cost of the video structured analysis system.

6. The method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection according to claim 1, characterized in that: Step 2 involves optimizing the parameter p m_init Select the algorithm as follows: Algorithm 1: Optimizing parameter p m_init Select Algorithm There is a certain coupling relationship between multiple modules, θ i It represents the degree of coupling between the current module and other modules. i The values are ordered from small to large to determine the decision priority of each module algorithm.

7. The method for software-hard collaborative optimization of a video structured analysis system based on parameter value selection according to claim 6, characterized in that: Assume that the initial parameter selection algorithm sets N target compression rate ranges Algorithm 1 gives the initial parameter optimization selection algorithm process, where: The rate-distortion cost is expressed as like The mth parameter in p m becomes The corresponding bit rate-computational complexity-algorithm accuracy cost is expressed as For each non-discrete parameter, it is discretized into K m is the number of candidate solutions for the mth parameter. By using a set of test sequences to calculate the integral area of the rate-computational complexity-algorithm accuracy cost curve within a certain rate range, the rate-distortion cost of all possible parameter combination solutions is estimated offline. Then, an iterative algorithm is used for parameter selection to converge to the predefined rate-computational complexity-algorithm accuracy cost.

Images