A method and device for adjusting engine parameters of an inference engine

Abstract

The present application discloses an engine parameter adjustment method for an inference engine, comprising: obtaining current cache information of a cache space used for caching data in an inference process in the inference engine, the current cache information comprising: the current data volume in the cache space, as well as the input rate and the output rate, using the output rate as the control amount for controlling the data volume of the cache space, accumulating the control amount, using the current data volume, the current input rate, and the current output rate, taking the accumulated control amount as a constraint, constructing an objective function for characterizing that the cache space does not overflow, obtaining an estimate of the control amount when the cache space does not overflow, determining a configured output rate with the smallest difference from the control amount estimate based on the control amount estimate, determining a configured engine parameter corresponding to the configured output rate based on the configured output rate, and adjusting the current engine parameter based on the configured engine parameter. The present application implements adaptive adjustment of the engine parameters of the inference engine, and improves the performance of the inference engine.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular, to a method and device for adjusting engine parameters of an inference engine. Background Art

[0002] An inference engine is a software component used to infer and accelerate artificial intelligence (AI) model tasks on hardware. Its working mode is usually a pipeline mode in which each task thread used to process different tasks processes tasks in sequence.

[0003] In the pipeline of the inference engine, the processing time of the task thread is related to the engine parameters associated with the task it processes. In the inference engine related technology, the engine parameters of the inference engine are usually fixed parameters during the inference process, which makes it difficult for the inference engine to have better performance due to changes in computing resources. Summary of the invention

[0004] The present invention provides an engine parameter adjustment method for an inference engine, so as to realize adaptive adjustment of engine parameters in the inference engine.

[0005] A first aspect of the present invention provides a method for adjusting engine parameters of an inference engine, the method comprising:

[0006] Obtaining current cache information of a cache space used to cache data in the inference process in the inference engine, wherein the current cache information includes: the current amount of data in the cache space, an input rate used to characterize the current rate at which data enters the cache space, and an output rate used to characterize the current rate at which data leaves the cache space,

[0007] The output rate is used as the control amount for controlling the amount of data in the cache space, and the control amount is accumulated. The current amount of data, the current input rate, and the current output rate are used to construct an objective function for characterizing that the cache space does not overflow, and the control amount estimate of the objective function when the cache space does not overflow is obtained.

[0008] According to the control amount estimation, determine the configuration output rate with the smallest difference from the control amount estimation,

[0009] According to the configured output rate, a configuration engine parameter corresponding to the configured output rate is determined, and the current engine parameter is adjusted according to the configuration engine parameter.

[0010] As a possible implementation, the inference engine includes a single-path inference engine.

[0011] As a possible implementation, the objective function is constructed as follows:

[0012] The control amount is sampled at the set sampling interval to obtain the control amount of each sampling point, and the absolute value of the control amount of each sampling point is accumulated to obtain the cumulative control amount.

[0013] Calculate the square value of the sum of the current data volume and the change data volume, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume is the product of the rate change between the current input rate and the current output rate and the sampling interval between adjacent sampling points;

[0014] As a possible implementation manner, obtaining a control amount estimation of the objective function when the cache space does not overflow includes:

[0015] The control amount that makes the objective function reach the minimum value is solved as the control amount estimate.

[0016] As a possible implementation, the inference engine is a multi-path inference engine that is inferred in parallel by multiple single-path inference engines, and each single-path inference engine corresponds to a cache space;

[0017] As a possible implementation, the obtaining of current cache information of a cache space in the inference engine for caching data in the inference process includes:

[0018] Get the current cache information of each cache space,

[0019] The current data volume of each cache space in the current cache information is used as an element in the vector to obtain a current data volume vector for representing the current data volume of each cache space of the multi-path inference engine.

[0020] The current input rate of each cache space in the current cache information is used as an element in the vector to obtain a current input rate vector for characterizing the current input rate of each cache space of the multi-path inference engine.

[0021] The current output rate of each cache space in the current cache information is used as an element in the vector to obtain a current output rate vector for characterizing the current output rate of each cache space of the multi-path inference engine.

[0022] As a possible implementation, the objective function is constructed as follows:

[0023] The control amount of each cache space is taken as an element in the vector to obtain the control amount vector for characterizing the control amount of each cache space of the multi-way inference engine.

[0024] The control vector is sampled at the set sampling interval to obtain the control vector of each sampling point, and the 1-norm of the control vector of each sampling point is calculated.

[0025] Calculate the 1-norm of the control quantity vector of each sampling point and add them up to get the cumulative control quantity.

[0026] Calculate the square value of the 2-norm of the current data volume vector and the change data volume vector, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume vector is: the product of the rate change vector between the current input rate vector and the current output rate vector and the sampling interval between adjacent sampling points;

[0027] As a possible implementation manner, obtaining a control amount estimation of the objective function when the cache space does not overflow includes:

[0028] Solve the control quantity vector that makes the objective function obtain the minimum value, and use each element in the control quantity vector as the control quantity estimation of the buffer space corresponding to the element;

[0029] As a possible implementation manner, determining, according to the control amount estimation, a configured output rate having a minimum difference from the control amount estimation includes:

[0030] For each inference engine in the multi-way inference engine,

[0031] According to the control amount estimation of the cache space of the inference engine, a configuration output rate with the smallest difference from the control amount estimation is determined as the configuration output rate of the cache space of the inference engine.

[0032] As a possible implementation, the configuration rate and configuration engine parameters are preconfigured.

[0033] in,

[0034] The configured output rate corresponds to the configured engine parameters one by one.

[0035] In the case of a single-path inference engine, the configured output rate is the first configured output rate,

[0036] In the case of multi-path parallel reasoning, the configured output rate is the second configured output rate, wherein the second configured output rate is the ratio of the first configured output rate to the number of parallel reasoning paths.

[0037] As a possible implementation, the multi-path inference engine is an inference engine for implementing an image target detection task, an image scaling thread for performing image scaling on a decoded image, and a target detection thread for performing target detection based on the scaled image.

[0038] in,

[0039] Each task is implemented by a single-path inference engine, each single-path inference engine includes: an image scaling thread for performing image scaling on a decoded image, and an object detection thread for performing object detection based on the scaled image,

[0040] The engine parameters of each single-path inference engine include: a detection resolution for controlling the clarity of the scaled image, and a detection frame rate for controlling the degree of frame extraction of the image frame, wherein the detection resolution is provided to the image scaling thread, and the detection frame rate is provided to the target detection thread;

[0041] The current input rate is the current video frame rate.

[0042] The current output rate is determined according to the set buffer space processing time, the time consumption of the image scaling thread at the current detection resolution, and the current frame extraction coefficient.

[0043] As a possible implementation, the change data amount vector is further modified in the following manner:

[0044] Multiply the diagonal matrix used to represent the number of targets detected by each target detection thread by the change data amount vector, and use the multiplication result as the change data amount vector.

[0045] Among them, the value of the element at the i-th row and i-th column in the diagonal matrix is ​​the number of targets detected by the i-th inference engine, and i is a natural number;

[0046] The determination according to the set buffer space processing time, the time consumption of the image scaling thread at the current detection resolution, and the current frame extraction coefficient includes:

[0047] The current output rate is obtained by calculating the ratio of the cache space processing time to the time consumed by the image scaling thread at the current detection resolution and the current frame extraction coefficient.

[0048] As a possible implementation manner, after determining the configuration engine parameter corresponding to the configured output rate according to the configured output rate, the method further includes:

[0049] Get the current occupancy information of hardware resources,

[0050] When the current occupancy of hardware resources is greater than the set occupancy threshold, based on the configuration information that corresponds one-to-one between the configuration rate and the configuration engine parameters, at least one of the configuration detection resolution and the configuration detection frame rate of each determined inference engine is reduced.

[0051] As a possible implementation manner, the configuration information based on the one-to-one correspondence between the configuration rate and the configuration engine parameter, reducing at least one of the configuration detection resolution and the configuration detection frame rate of each determined inference engine includes:

[0052] Sort the number of targets detected by each inference engine.

[0053] Prioritize reducing the configuration engine parameters of the inference engine with a small number of detected targets,

[0054] In the process of reducing the configuration engine parameters, in the order of reducing the configuration detection resolution first and then reducing the configuration detection frame rate, a configuration engine parameter that is smaller than the currently determined configuration engine parameter is selected in the configuration information to adjust the current engine parameter, and the configuration engine parameter of the inference engine is repeatedly reduced, so that the current occupation of the hardware resources after adjusting the current engine parameter according to the reduced configuration engine parameter is not greater than the occupation threshold.

[0055] If the configuration engine parameters cannot make the current occupancy of hardware resources less than the occupancy threshold when there is no room for downward adjustment, the configuration engine parameters of the next inference engine with fewer targets are reduced in order until the current occupancy of hardware resources is less than the occupancy threshold.

[0056] A second aspect of the present application provides an engine parameter adjustment device for an inference engine, the device comprising:

[0057] an acquisition module, used to acquire current cache information of a cache space used to cache data in an inference process in an inference engine, wherein the current cache information includes: a current amount of data in the cache space, an input rate used to characterize a current rate at which data enters the cache space, and an output rate used to characterize a current rate at which data leaves the cache space.

[0058] The control amount estimation module is used to use the output rate as the control amount for controlling the amount of data in the cache space, accumulate the control amount, use the current data amount, the current input rate, and the current output rate, and use the accumulated control amount as a constraint to construct an objective function for characterizing that the cache space does not overflow, and obtain the control amount estimation of the objective function when the cache space does not overflow.

[0059] The engine parameter determination module is used to determine the configured output rate with the smallest difference from the control amount estimation according to the control amount estimation, and determine the configuration engine parameters corresponding to the configured output rate according to the configured output rate,

[0060] The adjustment module is used to adjust the current engine parameters according to the determined configuration engine parameters.

[0061] A third aspect of the present application provides an inference engine system, which includes: a single-path inference engine or a multi-path inference engine, and the engine parameter adjustment device.

[0062] The present application provides an engine parameter adjustment method for an inference engine, which uses an output rate that characterizes the current rate at which data leaves a cache space as a control quantity, estimates the control quantity by using current cache information of a cache space in the inference engine used to cache data during an inference process, selects configuration engine parameters based on the control quantity estimate, and adjusts the current engine parameters according to the configuration engine parameters, so that the current engine parameters are at least adaptively adjusted according to the status of the cache space, thereby avoiding data loss due to insufficient cache space during the inference process, improving the performance of the inference engine, and taking into account the inference effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of an engine parameter adjustment method of the inference engine of the present application.

[0064] Figure 2 This is a schematic diagram of an inference engine for implementing a target detection task for a single-channel image in the first embodiment of the present invention.

[0065] Figure 3 A flowchart of a method for adjusting engine parameters of an inference engine according to an embodiment of the present invention is shown.

[0066] Figure 4 This is a schematic diagram of an inference engine for implementing parallel inference of target detection tasks for multiple images in the second embodiment of the present invention.

[0067] Figure 5 A flowchart of a method for adjusting engine parameters of the inference engine according to the second embodiment of the present invention is shown.

[0068] Figure 6 A schematic diagram of an engine parameter adjustment device of an inference engine according to an embodiment of the present application.

[0069] Figure 7 Another schematic diagram of the engine parameter adjustment device of the inference engine according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0071] The embodiment of the present application utilizes the current cache information of the cache space used to cache data during the inference process in the inference engine to configure the rate at which data leaves the cache space, determines the configuration engine parameters according to the configured rate, and adjusts the current engine parameters based on the configuration engine parameters, so that the current engine parameters are adjusted according to the cache status of the cache space, thereby realizing adaptive adjustment of the engine parameters of the inference engine and improving the performance of the inference engine.

[0072] See also Figure 1As shown, Figure 1 A flow chart of the engine parameter adjustment method of the inference engine of the present application. The method includes:

[0073] Step 101, obtaining the current cache information of the cache space used to cache data in the inference process in the inference engine,

[0074] Among them, the inference engine can be a single-path inference engine or a multi-path inference engine with multiple single-path parallel inference engines. The cache space can be the cache space used separately by each task thread of any one-path inference engine, or the cache space used jointly by each task thread of any one-path inference engine, or the cache space used jointly by the same task threads of different-path inference engines, or the cache space used separately by different task threads of different-path inference engines, or the cache space used jointly by different task threads of different-path inference engines, or a combination of the above situations, and the present application does not impose any restrictions on this.

[0075] The current cache information includes: the current amount of data in the cache space, the input rate used to characterize the current rate at which data enters the cache space, and the output rate used to characterize the current rate at which data leaves the cache space.

[0076] As an example, the input rate can be calculated based on the amount of data entering the cache within a set first time period, and the output rate can be calculated based on the amount of data leaving the cache within a set second time period. The first time period and the second time period can be the same or different, which can be determined based on the business implemented by the inference engine.

[0077] As an example, the current cache information may be obtained at a set time interval so as to sample the current cache information.

[0078] Take the example of using cache space for each inference engine in a multi-way inference engine. For example, n cache spaces are used in n-way inference engines, and one cache space is used for each way.

[0079] Get the current cache information of each cache space,

[0080] The current data volume of each cache space in the current cache information is used as an element in the vector to obtain a current data volume vector for representing the current data volume of each cache space of the multi-path inference engine.

[0081] The current input rate of each cache space in the current cache information is used as an element in the vector to obtain a current input rate vector for characterizing the current input rate of each cache space of the multi-path inference engine.

[0082] The current output rate of each cache space in the current cache information is used as an element in the vector to obtain a current output rate vector for characterizing the current output rate of each cache space of the multi-path inference engine.

[0083] Step 102, using the output rate as the control amount for controlling the amount of data in the cache space, accumulating the control amount, using the current amount of data, the current input rate, and the current output rate, and taking the accumulated control amount as a constraint, constructing an objective function for characterizing that the cache space does not overflow, and obtaining an estimate of the control amount of the objective function when the cache space does not overflow,

[0084] As an example, the objective function is constructed by maintaining a set change threshold with the amount of change in the cache space data. For example, the objective function is constructed with the amount of change in the cache space data being 0, as follows:

[0085] The control amount is sampled at the set sampling interval to obtain the control amount of each sampling point, and the absolute value of the control amount of each sampling point is accumulated to obtain the cumulative control amount.

[0086] Calculate the square value of the sum of the current data volume and the change data volume, and add the square value to the cumulative control volume as the objective function.

[0087] The amount of changed data is used to characterize the amount of data change in the cache space within the sampling interval between adjacent sampling points, and can be obtained by calculating the product of the rate change between the current input rate and the current output rate and the sampling interval between adjacent sampling points;

[0088] The obtaining of the control amount estimation of the objective function when the cache space does not overflow includes:

[0089] The control amount that makes the objective function reach the minimum value is solved as the control amount estimate.

[0090] Taking the example of each inference engine in a multi-path inference engine using cache space separately, the objective function is constructed as follows:

[0091] The control amount of each cache space is taken as an element in the vector to obtain the control amount vector for characterizing the control amount of each cache space of the multi-way inference engine.

[0092] The control vector is sampled at the set sampling interval to obtain the control vector of each sampling point, and the 1-norm of the control vector of each sampling point is calculated.

[0093] Calculate the 1-norm of the control quantity vector of each sampling point and add them up to get the cumulative control quantity.

[0094] Calculate the square value of the 2-norm of the current data volume vector and the change data volume vector, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume vector is: the product of the rate change vector between the current input rate vector and the current output rate vector and the sampling interval between adjacent sampling points;

[0095] The control quantity vector that makes the objective function obtain the minimum value is solved, and each element in the control quantity vector is used as the control quantity estimation of the cache space corresponding to the element.

[0096] Step 103, based on the control amount estimation, determine the configured output rate with the smallest difference from the control amount estimation,

[0097] As an example, a configured output rate closest to the control amount estimate is selected from the configured output rate set.

[0098] Taking the example of each inference engine in a multi-path inference engine using cache space separately, for each inference engine in the multi-path inference engine, based on the control quantity estimate of the cache space of the inference engine, the configuration output rate with the smallest difference from the control quantity estimate is determined as the cache space configuration output rate of the inference engine.

[0099] Step 104: Determine a configuration engine parameter corresponding to the configuration output rate according to the configuration output rate, and adjust the current engine parameter according to the configuration engine parameter.

[0100] As an example, the configured rate corresponds to the configured engine parameter one by one to serve as the configuration information of the inference engine. The configuration information can be configured through an external interface to meet application requirements in different application scenarios.

[0101] In the case of a single-path inference engine, the configured output rate is the first configured output rate,

[0102] In the case of multi-path parallel reasoning, the configured output rate is the second configured output rate, wherein the second configured output rate is the ratio of the first configured output rate to the number of parallel reasoning paths.

[0103] As an example, the above steps 101 to 104 can be implemented by a scheduling thread independent of the inference engine, or can be implemented by a setting thread in the inference engine, and this application does not impose any restrictions on this.

[0104] The embodiment of the present application adjusts the engine parameters by the current status of the cache space, which not only realizes the maximum utilization of hardware resources, but also avoids data loss during the reasoning process, thereby taking into account the reasoning effect of the reasoning engine. Since the adjustment process does not require the parameters of each task thread in the reasoning engine itself, the engine parameter adjustment method of the reasoning engine of the present application is not only applicable to different platforms, but also to different businesses.

[0105] To facilitate understanding of the embodiments of the present application, the following explanation is given using an inference engine for implementing an image target detection task as an example. It should be understood that the present application is not limited to the inference engine for this specific task, and inference engines for implementing other tasks may also be applicable.

[0106] As large AI models are widely deployed in various application scenarios, the requirements of inference engines for hardware computing resources are getting higher and higher. Taking image target detection tasks as an example, the increase in detection rate and accuracy will lead to an increase in the use of hardware computing resources by the inference engine. When the load of the inference engine exceeds the maximum hardware resources, it will cause frequent image frame loss and cause false detection.

[0107] The inference engines currently used to implement image target detection tasks usually execute fixed engine parameters. This mode will cause insufficient hardware resources in high-load situations, resulting in continuous frame loss in the inference engine, which will lead to a sharp deterioration in target detection results.

[0108] Embodiment 1

[0109] See also Figure 2 As shown, Figure 2 This is a schematic diagram of an inference engine for implementing a target detection task for a single-channel image in the first embodiment of the present invention.

[0110] The pipeline of the inference engine includes: a decoding thread for decoding image frames, an image scaling thread for scaling the decoded image frames, a target detection thread for detecting targets based on scaled image frames, a target tracking thread for tracking detected targets, and a post-processing thread for information integration of detection results, wherein the pipeline is composed of the decoding thread, the image scaling thread, the target detection thread, and the post-processing thread in sequence. In order to alleviate the frame loss caused by thread time consumption, a cache queue for caching data is set for the pipeline. In this embodiment, the data cached by the cache queue is the decoded image frame decoded by the decoding thread. It should be understood that the data cached by the cache queue is not limited to this, and data of other threads can also be cached.

[0111] In the pipeline of this embodiment, the engine parameters include: a detection resolution for controlling the clarity of image scaling, and a detection frame rate for controlling the degree of extraction of image frames. The processing time of the image scaling thread is related to the detection resolution. The larger the detection resolution, the longer the processing time of the image scaling thread, and the better the detection rate and accuracy rate without causing frame loss; the processing time of the target detection thread is related to the detection frame rate. The larger the detection frame rate, the more frames are extracted for the scaled image frame, and the longer the processing time of the target detection thread, and the better the detection rate and accuracy rate without causing frame loss.

[0112] In order to take into account both the detection rate and the precision rate while avoiding frame loss, this embodiment adjusts the engine parameters during the inference process by scheduling threads.

[0113] See also Figure 3 As shown, Figure 3FIG. 1 is a flow chart of a method for adjusting engine parameters of an inference engine according to an embodiment of the present invention. The method includes, on the scheduling thread side,

[0114] Step 301, set N different configuration detection frame rates, M different configuration detection resolutions, and establish a corresponding relationship between the configuration output rate of the cache queue and the configuration detection frame rate and configuration detection resolution as configuration information of the inference engine.

[0115] The configured output rate is used to characterize the rate of extracting data from the cache. Since the decoded data of the decoding thread is cached according to the image frame as the minimum granularity, the configured output rate in this embodiment characterizes the frame rate of extracting image frames from the cache, that is, the number of image frames extracted from the cache queue within a set unit time. The configured output rate, configured detection frame rate, and configured detection resolution correspond to each other.

[0116] As an example, the detection frame rate is configured to be N different levels, and the detection resolution is configured to be M different levels, where N and M can be the same or different, and both N and M are natural numbers. Table 1 is a schematic diagram of the corresponding relationship between the configured output rate, the configured detection frame rate, and the configured detection resolution.

[0117]

[0118] In step 302, the current detection frame rate and the current detection resolution are respectively initialized to the maximum values ​​in the configuration parameters. That is, the maximum configured detection frame rate among N different configured detection frame rates is used as the initial value of the current detection frame rate, and the maximum configured detection resolution among M different configured detection resolutions is used as the initial value of the current detection resolution.

[0119] Step 303, obtain the total number of current image frames in the cache queue, as well as the current input rate and the current output rate, wherein the input rate represents the frame rate of the image frames stored in the cache queue, that is, the number of image frames stored in the cache queue within a set unit time length.

[0120] In this embodiment, since the decoding thread is the first thread in the pipeline, it is not affected by the previous thread, so the input rate can be a constant value.

[0121] Step 304, based on the total number of image frames currently in the cache queue, the current input rate, and the current output rate, take the output rate as the control variable for controlling the amount of data in the cache space, construct an objective function, and use the objective function to estimate the control variable to obtain a control variable estimate, which represents the rate estimate for extracting image frames from the cache queue, that is, the estimated number of image frames extracted from the cache queue within a set unit time length.

[0122] In this step, the goal of adjusting the engine parameters is not general, but to control the cache queue from overflowing, so it can be modeled as a control problem for the amount of cached data in the cache queue.

[0123] For the cache queue in the inference engine, the amount of data stored in the cache queue may not be a stable value, while the amount of data taken out of the cache queue is a controllable amount.

[0124] Since the amount of data change in the cache queue is the difference between the amount of data entering the cache queue and the amount of data leaving the cache queue, for any cache queue, the change rate of the cache queue can be expressed as the following formula (1):

[0125] in, Indicates the rate at which data in the cache queue changes.

[0126] v in (t) represents the queue entry rate, v out (t) represents the dequeue rate, which is equivalent to the queue processing rate.

[0127] In this embodiment, v in (t) is a constant value v, which is usually the current video frame rate; the current dequeue rate is determined according to the set cache space processing time, the time consumed by the image scaling thread at the current detection resolution, and the current frame extraction coefficient. As an example, the current dequeue rate is the ratio of the cache space processing time to the time consumed by the image scaling thread at the current detection resolution and the current frame extraction coefficient, v out (t) can be expressed as the following formula (2):

[0128]

[0129] in, Indicates the unit processing time of the set cache queue, which is generally 1000ms. m(t) indicates the time consumption of the image scaling thread at the current detection resolution. It is the current frame extraction coefficient, which is used to represent the number of frames extracted every certain number of frames.

[0130] In order to control the outflow rate of data in the queue, the outflow rate is used as the control variable, and the queue control variable is defined as u(t). The state equation of the queue can be expressed as the following formula (3):

[0131]

[0132] Use the Euler method to discretize the state equation above:

[0133]

[0134] Wherein, T is the sampling interval between adjacent sampling points, which is usually a fixed value, q(k+1) is the amount of data in the cache queue at sampling point k+1, q(k) is the amount of data in the cache queue at sampling point k, u(k) is the control amount at sampling point k, and the sampling time interval between sampling point k+1 and sampling point k is T. Preferably, the sampling time interval is less than the unit processing time so that as much sample data as possible can be obtained within the unit processing time.

[0135] As an example, taking the control of the queue length change to 0 as the goal, the mathematical expression of the objective function is constructed as follows:

[0136]

[0137] Wherein, k is a sampling point, for example, it may correspond to the time point of obtaining the current cache information, , K is the total number of accumulated sampling points; q(K) is the current total data volume of the queue at sampling point K, which is the total number of currently decoded image frames in the cache queue in this embodiment; (v-u(K))T is the change in the current data volume of the queue within the sampling interval, u(K) is the control volume of the cache queue at sampling point K, that is, the current output rate of the cache queue, which represents the output frame rate of the cache queue at sampling point K in this embodiment; u(k) is the control volume of the cache queue at sampling point k, which represents the output frame rate of the cache queue at sampling point k in this embodiment;

[0138] is the accumulation of control quantities at each sampling point, i.e., the cumulative control quantity, which is used to regularize u(k).

[0139] To form the constraint term of the objective function;

[0140] u(K) and u(k) can be obtained according to formula (2), that is: u(k) = v out (k) = C / (m(k)s), where v out (k) represents the dequeue rate at sampling point k, C represents the unit processing time of the set cache queue, m(k) represents the time consumption of the image scaling thread at the detection resolution at sampling point k, and s is the current frame extraction coefficient.

[0141]

[0142] Solve the control amount that makes the objective function reach the minimum value and obtain the control amount estimate.

[0143] Step 305: determine the configuration output rate that is closest to the control amount estimate in the configuration information, and determine the configuration detection frame rate and configuration detection resolution corresponding to the configuration output rate according to the determined configuration output rate.

[0144] Step 306: According to the determined configured detection frame rate and configured detection resolution, the current detection frame rate and the current detection resolution are adjusted to the determined configured detection frame rate and configured detection resolution.

[0145] Step 307, return to step 303 until the reasoning is completed.

[0146] This embodiment estimates the control amount through the total data volume of the queue, the current output rate and the control amount at each sampling point, so that the data volume in the cache queue changes to 0, and matches the corresponding configuration detection resolution and detection frame rate according to the control volume estimation, so that the engine parameters are dynamically adjusted as the data volume of the cache queue changes, thereby avoiding the loss of image frames, and improving the utilization rate of hardware resources while obtaining accurate target detection results.

[0147] Embodiment 2

[0148] See also Figure 4 As shown, Figure 4 This is a schematic diagram of an inference engine for implementing parallel inference of target detection tasks for multiple images in the second embodiment of the present invention.

[0149] The inference engine includes multiple pipelines, each of which is used to implement the target detection task of one image. In order to alleviate the frame loss caused by thread time consumption, a cache queue is set for each pipeline. In this embodiment, the data cached by the cache queue is the decoded image frame decoded by the decoding thread. It should be understood that the data cached by the cache queue is not limited to this, and data of other threads can also be cached. The scheduling thread adjusts the engine parameters of each pipeline respectively.

[0150] For multi-channel parallel inference engines, since the image bitstreams sent to each inference engine are not necessarily the same, it is necessary to select more appropriate engine parameters according to the bitstream. The adjustment method is more complicated than single-channel inference. In addition, multi-channel concurrency will cause competition for hardware resources. In the case of insufficient hardware resources, the inference time will become uncontrollable. Therefore, it is better to introduce the observation value of hardware resources into the adjustment process.

[0151] Compared with the reasoning time of a single-path reasoning engine, multi-path parallelism will lead to increased reasoning time. Given that the reasoning time and the number of parallel paths can be approximately linearly related, the configured output rate in the single-path case can be corrected to serve as the configured output rate in the multi-path case. In other words, when only one inference engine in the multi-path parallel reasoning engine performs reasoning, the configured output rate is the first configured output rate in the first configured output rate set; when more than one inference engine in the multi-path parallel reasoning engine performs reasoning, the configured output rate is the second configured output rate in the second configured output rate set, wherein the second configured output rate is the ratio of the first configured output rate to the number of parallel paths. It can be expressed as:

[0152]

[0153] Where I is the number of parallel paths, is the second configured output rate, v out (t) is the first configured output rate.

[0154] For multi-channel parallel scenarios, formula (4) is converted into vector form for easy calculation. Specifically, the total number of frames buffered by each queue can be expressed as a vector, and the mathematical formula is:

[0155]

[0156] Wherein, Q(k) is the current data amount vector in the cache space of the multi-channel inference engine at sampling point k. In this embodiment, it represents the current total frame number vector at sampling point k, q i (k) is the current total number of frames in the cache queue of the i-th inference engine at sampling point k, [ ] T Represents a transpose operation, i is less than or equal to the number of parallel paths I.

[0157] The control quantity can be converted into a control quantity vector, and the mathematical formula is:

[0158]

[0159] Among them, U(k) is the control vector of the cache space of the multi-way inference engine at sampling point k, and u i (k) is the control amount of the cache queue of the i-th inference engine at sampling point k, [ ] T Represents a transpose operation, i is less than or equal to the number of parallel paths I.

[0160] The objective function in the multi-path case can be expressed as:

[0161]

[0162] in, K is the total number of accumulated sampling points; Q(K) is the current total number of frames in the cache space of the multi-way inference engine at sampling point K, U(K) is the control vector of the cache space of the multi-way inference engine at sampling point K, that is, the current output rate vector at sampling point K, V is the current input rate vector, which can be expressed as a constant vector, (V- U(K))×T is the change vector of the cache space of the multi-way inference engine within the sampling interval,

[0163] represents the square of the 2-norm of a vector,

[0164] represents the 1-norm of a vector,

[0165] Represents the accumulation of the 1-norm values ​​of the control vectors of each sampling point to obtain the cumulative control amount.

[0166] Furthermore, since the number of targets detected by the target detection threads in each inference engine is different, the number of detected targets can be used as the control constraint of the objective function to allocate different engine parameters to each inference engine. Formula (6) can be modified as follows:

[0167]

[0168] Among them, W(K) is a diagonal matrix composed of the number of targets detected by each inference engine at the acquisition point K, and the element value at the i-th row and i-th column in the diagonal matrix is ​​the number of targets detected by the i-th inference engine.

[0169] See also Figure 5 As shown, Figure 5 FIG. 1 is a flow chart of a method for adjusting engine parameters of the second inference engine of the present embodiment. The method includes, on the scheduling thread side,

[0170] Step 501: For each inference engine, establish a corresponding relationship between the configured cache frame rate of its cache queue and the configured detection frame rate and configured detection resolution.

[0171] in,

[0172] The configured output rate, configured detection frame rate, and configured detection resolution configured for each inference engine may be the same or different.

[0173] To reduce complexity, in this embodiment, the configuration information of each inference engine is the same, as shown in Table 2, which is the configuration information of the engine parameters.

[0174]

[0175] Given that the number of current parallel reasoning paths in a multi-path reasoning engine may change dynamically, for example, the reasoning of a certain reasoning engine ends, or the number of reasoning engines used changes, in order to improve the performance of the reasoning engine, the second configuration output rate can be updated in real time according to the current number of parallel reasoning paths.

[0176] Step 502: Initialize the current engine parameters of each inference engine. The current engine parameters include detection frame rate and detection resolution.

[0177] The current engine parameters of the inference engines of each path may be the same or different. In this embodiment, the maximum detection frame rate and the maximum detection resolution are initialized.

[0178] Step 503, obtaining the total number of current image frames in each cache queue, as well as the current input rate and the current output rate.

[0179] As an example, the total number of current image frames of each cache queue in the current cache information is used as an element in the vector to obtain a current data volume vector representing the total number of current image frames of each cache queue of the multi-channel inference engine.

[0180] The current inbound rate of each cache queue in the current cache information is used as an element in the vector to obtain a current inbound rate vector for characterizing the current inbound rate of each cache queue of the multi-path inference engine.

[0181] The current output rate of each cache queue in the current cache information is used as an element in the vector to obtain a current output rate vector for characterizing the current output rate of each cache queue of the multi-path inference engine.

[0182] Step 504, using the current data volume vector, the current input rate vector, and the current output rate vector, and taking the control quantity vector as a constraint term, an objective function is constructed according to formula (6) or (7), and the control quantity vector is solved when the objective function obtains the minimum value, thereby obtaining an estimate of the control quantity vector.

[0183] Step 505: for each inference engine, according to the control quantity vector estimation, based on the configuration information, determine the configuration output rate that is closest to the control quantity estimation of the path, and according to the determined configuration output rate, determine the configuration detection frame rate and configuration detection resolution corresponding to the configuration output frame rate.

[0184] Step 506: To ensure that the inference time is controllable, further obtain the current occupancy information of the hardware resources, including GPU utilization and / or CPU utilization.

[0185] Step 507: determine whether the current occupancy of the hardware resources exceeds a set occupancy threshold, for example, whether the GPU utilization and / or the CPU utilization exceeds a utilization threshold.

[0186] If yes, then the configuration engine parameters determined in step 505 are reduced, for example, at least one of the detection frame rate and the configuration detection resolution, to reduce the current utilization of the hardware resources.

[0187] As an example, the number of targets currently detected by each inference engine is sorted, and the configuration engine parameters of the inference engine with a small number of targets are reduced first. The configuration engine parameters are reduced in the order of first reducing the configuration detection resolution and then reducing the configuration detection frame rate. The current engine parameters are adjusted according to the reduced configuration engine parameters, and the configuration engine parameters are repeatedly reduced until the current occupancy of hardware resources is less than the occupancy threshold. If the configuration engine parameters have no room for downward adjustment and still cannot make the current occupancy of hardware resources less than the occupancy threshold, then the configuration engine parameters of the next inference engine with a small number of targets are reduced according to the sort, and the configuration engine parameters of the inference engines with a small number of targets in the sort are reduced one by one in this way until the current occupancy of hardware resources is less than the occupancy threshold.

[0188] As an example, the configuration engine parameter is lowered in the following manner: a configuration engine parameter that is smaller than the configuration engine parameter is selected from the configuration information of the engine parameter to adjust the current engine parameter.

[0189] By lowering the configuration engine parameters, hardware resources can be quickly released while taking into account the reasoning effect of the inference engine.

[0190] Otherwise, the current detection frame rate and detection resolution are adjusted according to the configuration detection frame rate and configuration detection resolution determined in step 505.

[0191] Step 508, return to step 503, until the reasoning of each reasoning engine is completed.

[0192] This embodiment obtains the current hardware resource information and the cache queue status information of each inference engine to estimate the control amount of the cache queue of each inference engine, and then adjusts its engine parameters based on the control amount estimation of each inference engine, so that the engine parameters can be adaptively adjusted, which is conducive to maximizing the utilization of hardware resources, ensuring the inference effect, and improving the overall performance of the inference engine. This embodiment uses a scheduling thread to summarize the cache information and current hardware resource information of each inference engine, which is conducive to the unified management of the engine parameters of each inference engine.

[0193] See also Figure 6 As shown, Figure 6 A schematic diagram of an engine parameter adjustment device of an inference engine according to an embodiment of the present application. As an example, the device includes a scheduling thread module, and the scheduling thread module includes:

[0194] an acquisition module, used to acquire current cache information of a cache space used to cache data in an inference process in an inference engine, wherein the current cache information includes: a current amount of data in the cache space, an input rate used to characterize a current rate at which data enters the cache space, and an output rate used to characterize a current rate at which data leaves the cache space.

[0195] The control amount estimation module is used to use the output rate as the control amount for controlling the amount of data in the cache space, accumulate the control amount, use the current data amount, the current input rate, and the current output rate, and use the accumulated control amount as a constraint to construct an objective function for characterizing that the cache space does not overflow, and obtain the control amount estimation of the objective function when the cache space does not overflow.

[0196] The engine parameter determination module is used to determine the configured output rate with the smallest difference from the control amount estimation according to the control amount estimation, and determine the configuration engine parameters corresponding to the configured output rate according to the configured output rate,

[0197] The adjustment module is used to adjust the current engine parameters according to the determined configuration engine parameters to provide them to the threads in the reasoning engine.

[0198] As an example, the acquisition module further includes: for acquiring current occupancy information of hardware resources,

[0199] The engine parameter determination module also includes: when the current occupancy of the hardware resources is greater than the set occupancy threshold, based on the configuration information of the one-to-one correspondence between the configuration rate and the configuration engine parameter, reducing the configuration engine parameters of each determined inference engine.

[0200] See also Figure 7 As shown, Figure 7 This is another schematic diagram of an engine parameter adjustment device for an inference engine according to an embodiment of the present application. The device includes a memory and a processor. The memory stores a computer program. The processor is configured to execute the computer program to implement the steps of the engine parameter adjustment method for the inference engine according to the embodiment of the present application.

[0201] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0202] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0203] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the engine parameter adjustment method of the inference engine described in the embodiment of the present application are implemented.

[0204] As for the apparatus / network-side device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0205] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for adjusting engine parameters of an inference engine, characterized in that: The method includes: Obtaining current cache information of a cache space used to cache data in the inference process in the inference engine, wherein the current cache information includes: the current amount of data in the cache space, an input rate used to characterize the current rate at which data enters the cache space, and an output rate used to characterize the current rate at which data leaves the cache space, The output rate is used as the control amount for controlling the amount of data in the cache space, and the control amount is accumulated. The current amount of data, the current input rate, and the current output rate are used to construct an objective function for characterizing that the cache space does not overflow, and the control amount estimate of the objective function when the cache space does not overflow is obtained. According to the control amount estimation, determine the configuration output rate with the smallest difference from the control amount estimation, Determine a configuration engine parameter corresponding to the configuration output rate according to the configuration output rate, and adjust the current engine parameter according to the configuration engine parameter, in, The objective function is constructed as follows: For a single-path inference engine, The control amount is sampled at the set sampling interval to obtain the control amount of each sampling point, and the absolute value of the control amount of each sampling point is accumulated to obtain the cumulative control amount. Calculate the square value of the sum of the current data volume and the change data volume, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume is the product of the rate change between the current input rate and the current output rate and the sampling interval between adjacent sampling points; For a multi-path inference engine that is inferred in parallel by multiple single-path inference engines, The control amount of each cache space is taken as an element in the vector to obtain a control amount vector for characterizing the control amount of each cache space of the multi-way inference engine. The control vector is sampled at the set sampling interval to obtain the control vector of each sampling point, and the 1-norm of the control vector of each sampling point is calculated. Calculate the 1-norm of the control quantity vector of each sampling point and add them up to get the cumulative control quantity. Calculate the square value of the 2-norm of the current data volume vector and the change data volume vector, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume vector is: the product of the rate change vector between the current input rate vector and the current output rate vector and the sampling interval between adjacent sampling points.

2. The engine parameter adjustment method according to claim 1, characterized in that: The inference engine comprises a single-path inference engine, The obtaining of the control amount estimation of the objective function when the cache space does not overflow includes: The control amount that makes the objective function reach the minimum value is solved as the control amount estimate.

3. The engine parameter adjustment method according to claim 1, characterized in that: The inference engine is a multi-path inference engine that is inferred in parallel by multiple single-path inference engines, and each single-path inference engine corresponds to a cache space; The obtaining of current cache information of a cache space in the inference engine for caching data in the inference process includes: Get the current cache information of each cache space, The current data volume of each cache space in the current cache information is used as an element in the vector to obtain a current data volume vector for representing the current data volume of each cache space of the multi-way inference engine. The current input rate of each cache space in the current cache information is used as an element in the vector to obtain a current input rate vector for characterizing the current input rate of each cache space of the multi-path inference engine. The current output rate of each cache space in the current cache information is used as an element in the vector to obtain a current output rate vector for characterizing the current output rate of each cache space of the multi-path inference engine.

4. The engine parameter adjustment method according to claim 3, characterized in that: The obtaining of the control amount estimation of the objective function when the cache space does not overflow includes: Solve the control quantity vector that makes the objective function obtain the minimum value, and use each element in the control quantity vector as the control quantity estimation of the buffer space corresponding to the element; The step of determining, based on the control amount estimation, a configured output rate having the smallest difference from the control amount estimation includes: For each inference engine in the multi-way inference engine, According to the control amount estimation of the cache space of the inference engine, a configuration output rate with the smallest difference from the control amount estimation is determined as the configuration output rate of the cache space of the inference engine.

5. The engine parameter adjustment method according to claim 1, characterized in that: The configuration rate and configuration engine parameters are pre-configured. in, The configured output rate corresponds to the configured engine parameters one by one. In the case of a single-path inference engine, the configured output rate is the first configured output rate, In the case of multi-path parallel reasoning, the configured output rate is the second configured output rate, wherein the second configured output rate is the ratio of the first configured output rate to the number of parallel reasoning paths.

6. The engine parameter adjustment method according to claim 4, characterized in that: The multi-path inference engine is an inference engine for implementing an image target detection task, an image scaling thread for performing image scaling on a decoded image, and a target detection thread for performing target detection based on the scaled image. in, Each task is implemented by a single-path inference engine, each single-path inference engine includes: an image scaling thread for performing image scaling on a decoded image, and an object detection thread for performing object detection based on the scaled image, The engine parameters of each single-path inference engine include: a detection resolution for controlling the clarity of the scaled image, and a detection frame rate for controlling the degree of frame extraction of the image frame, wherein the detection resolution is provided to the image scaling thread, and the detection frame rate is provided to the target detection thread; The current input rate is the current video frame rate. The current output rate is determined according to the set buffer space processing time, the time consumption of the image scaling thread at the current detection resolution, and the current frame extraction coefficient.

7. The engine parameter adjustment method according to claim 6, characterized in that: The change data volume vector is further modified in the following manner: Multiply the diagonal matrix used to represent the number of targets detected by each target detection thread by the change data amount vector, and use the multiplication result as the change data amount vector. Among them, the value of the element at the i-th row and i-th column in the diagonal matrix is ​​the number of targets detected by the i-th inference engine, and i is a natural number; The determination according to the set buffer space processing time, the time consumption of the image scaling thread at the current detection resolution, and the current frame extraction coefficient includes: The current output rate is obtained by calculating the ratio of the cache space processing time to the time consumed by the image scaling thread at the current detection resolution and the current frame extraction coefficient.

8. The engine parameter adjustment method according to claim 6, characterized in that: After determining the configuration engine parameter corresponding to the configured output rate according to the configured output rate, the method further includes: Get the current occupancy information of hardware resources, When the current occupancy of hardware resources is greater than the set occupancy threshold, based on the configuration information that corresponds one-to-one between the configuration rate and the configuration engine parameters, at least one of the configuration detection resolution and the configuration detection frame rate of each determined inference engine is reduced.

9. The engine parameter adjustment method according to claim 8, characterized in that: The reducing at least one of the configuration detection resolution and the configuration detection frame rate of each determined inference engine based on the configuration information corresponding to the configuration rate and the configuration engine parameter comprises: Sort the number of targets detected by each inference engine. Prioritize reducing the configuration engine parameters of the inference engine with a small number of detected targets, In the process of reducing the configuration engine parameters, in the order of reducing the configuration detection resolution first and then reducing the configuration detection frame rate, a configuration engine parameter that is smaller than the currently determined configuration engine parameter is selected in the configuration information to adjust the current engine parameter, and the configuration engine parameter of the inference engine is repeatedly reduced, so that the current occupation of the hardware resources after adjusting the current engine parameter according to the reduced configuration engine parameter is not greater than the occupation threshold. If the configuration engine parameters cannot make the current occupancy of hardware resources less than the occupancy threshold when there is no room for downward adjustment, the configuration engine parameters of the next inference engine with fewer targets are reduced in order until the current occupancy of hardware resources is less than the occupancy threshold.

10. An engine parameter adjustment device for an inference engine, characterized in that: The device includes: an acquisition module, used to acquire current cache information of a cache space used to cache data in an inference process in an inference engine, wherein the current cache information includes: a current amount of data in the cache space, an input rate used to characterize a current rate at which data enters the cache space, and an output rate used to characterize a current rate at which data leaves the cache space. The control amount estimation module is used to use the output rate as the control amount for controlling the amount of data in the cache space, accumulate the control amount, use the current data amount, the current input rate, and the current output rate, and use the accumulated control amount as a constraint to construct an objective function for characterizing that the cache space does not overflow, and obtain the control amount estimation of the objective function when the cache space does not overflow. The engine parameter determination module is used to determine the configured output rate with the smallest difference from the control amount estimation according to the control amount estimation, and determine the configuration engine parameters corresponding to the configured output rate according to the configured output rate, An adjustment module, used to adjust current engine parameters according to the determined configuration engine parameters; in, The objective function is constructed as follows: For a single-path inference engine, The control amount is sampled at the set sampling interval to obtain the control amount of each sampling point, and the absolute value of the control amount of each sampling point is accumulated to obtain the cumulative control amount. Calculate the square value of the sum of the current data volume and the change data volume, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume is the product of the rate change between the current input rate and the current output rate and the sampling interval between adjacent sampling points; For a multi-path inference engine that is inferred in parallel by multiple single-path inference engines, The control amount of each cache space is taken as an element in the vector to obtain the control amount vector for characterizing the control amount of each cache space of the multi-way inference engine. The control vector is sampled at the set sampling interval to obtain the control vector of each sampling point, and the 1-norm of the control vector of each sampling point is calculated. Calculate the 1-norm of the control quantity vector of each sampling point and add them up to get the cumulative control quantity. Calculate the square value of the 2-norm of the current data volume vector and the change data volume vector, and add the square value to the cumulative control amount to obtain the objective function, where the change data volume vector is: the product of the rate change vector between the current input rate vector and the current output rate vector and the sampling interval between adjacent sampling points.

11. An inference engine system, characterized in that: The device comprises: a single-path inference engine or a multi-path inference engine, and the engine parameter adjustment device as claimed in claim 10.

Images