A method and device for adjusting frame rate of frame extraction

Abstract

This invention provides a frame extraction frame rate adjustment method and apparatus. The method includes: obtaining the recognition time of a first algorithm on the extracted current video frame; obtaining the CPU utilization rate of the central processing unit; and adjusting the frame extraction frame rate according to the recognition time and the CPU utilization rate. This method can solve the problem of fixed frame rate processing by the algorithm in related technologies, which cannot be applied to different scenarios and has poor flexibility. In scenarios with fewer targets (low pressure), the number of frames sent can be increased to improve the algorithm effect. In scenarios with more targets (high pressure), the number of frames sent can be reduced to ensure smooth picture. This method can be flexibly applied to different scenarios.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a frame rate adjustment method and device. Background Art

[0002] In the field of monitoring, in addition to image acquisition, it is also necessary to use algorithms to identify targets in the image. The hardware carriers for algorithm calculations include: CPU, GPU, and other dedicated hardware such as DSP, NN IE, NPU, etc. Usually, one of the above algorithm hardware carriers is used to perform some simple target detection (such as faces, vehicles); then the CPU performs execution business, which may also use algorithms (such as target optimization, linkage, data comparison, etc.). Due to the performance (i.e. computing power) and cost of these computing hardware, it is impossible to perform algorithm analysis on every frame of the collected image, so there is a frame extraction process.

[0003] Scenarios that require intelligent algorithms mainly include: perimeter (human body, motor vehicle, non-motor vehicle and other tripwires, area intrusion), video structuring, face recognition and other business scenarios.

[0004] In the process of business processing, in order to avoid lag caused by insufficient performance, the frame rate of algorithm processing is fixed in the relevant technology. Usually, the frame rate of the algorithm processing unit is selected between 1 / 5 and 1 / 2 frames. When selecting, hardware performance, application scenarios, etc. are comprehensively considered. The higher the frame rate, the better the effect, which depends on hardware performance and application scenarios. The frame rate of fixed algorithm processing has poor flexibility in dealing with different scenarios.

[0005] There is no solution yet to the problem that the frame rate processed by the fixed algorithm in the related technology cannot be applied to different scenarios and has poor flexibility. Summary of the invention

[0006] The embodiments of the present invention provide a frame rate adjustment method and device for frame extraction, so as to at least solve the problem that the frame rate processed by a fixed algorithm in the related art cannot be applied to different scenarios and has poor flexibility.

[0007] According to an embodiment of the present invention, a frame rate adjustment method is provided, comprising:

[0008] Obtaining the time consumed by the first algorithm to recognize the extracted current video frame;

[0009] Get the CPU usage of the central processing unit;

[0010] The frame rate of frame extraction is adjusted according to the recognition time consumption and the CPU occupancy rate.

[0011] Optionally, the method further comprises: after obtaining the recognition time consumed by the first algorithm for the extracted current video frame, obtaining the processing time consumed by the second algorithm for processing the recognition result of the first algorithm;

[0012] Adjusting the frame rate of frame extraction according to the recognition time consumption and the CPU occupancy rate includes: adjusting the frame rate of frame extraction according to the recognition time consumption, the processing time consumption and the CPU occupancy rate.

[0013] Optionally, adjusting the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption and the CPU occupancy rate includes:

[0014] If the recognition time is greater than the ratio of the preset first minimum computing power threshold to the frame rate of frame extraction or the processing time is greater than the ratio of the preset second minimum computing power threshold to the frame rate of frame extraction, reducing the frame rate of frame extraction at a first predetermined interval;

[0015] If the recognition time is less than the ratio of the preset first maximum computing power threshold to the frame rate, or the processing time is greater than the ratio of the preset second maximum computing power threshold to the frame rate, the frame rate is adjusted according to the recognition time, the processing time, the a core occupancy rate and the b core occupancy rate, wherein the CPU occupancy rate includes the a core occupancy rate and the b core occupancy rate.

[0016] Optionally, adjusting the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption, the a-core occupancy rate, and the b-core occupancy rate includes:

[0017] If the current occupancy rate of the a-core is less than the preset a-core idle threshold and the current occupancy rate of the b-core is less than the preset b-core idle threshold, increasing the frame rate at a second predetermined interval;

[0018] If the current occupancy rate of the a core is less than the a core idle threshold and the current occupancy rate of the b core is not less than the b core idle threshold, or the current occupancy rate of the a core is not less than the a core idle threshold and the current occupancy rate of the b core is less than the b core idle threshold, and both the first algorithm and the second algorithm are in the a core of the CPU, and the current occupancy rate of the a core is less than the a core idle threshold, increase the frame rate at the second predetermined interval;

[0019] If the current occupancy of core a is greater than the preset busy threshold of core a or the current occupancy of core b is greater than the preset busy threshold of core b, obtain the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm, and adjust the frame rate of the frame extraction according to the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm.

[0020] Optionally, adjusting the frame rate of the frame extraction according to the CPU occupancy rate of the first algorithm and the CPU occupancy rate of the second algorithm includes:

[0021] If the first algorithm is in the a core of the CPU and the second algorithm is in the b core of the CPU, when the CPU occupancy rate of the first algorithm is greater than the a core idle threshold or the CPU occupancy rate of the second algorithm is greater than the b core idle threshold, the frame rate of the frame extraction is reduced at the first preset interval;

[0022] If the first algorithm and the second algorithm are both in the a core of the CPU, when the sum of the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm is greater than the a core busy threshold, the frame rate is reduced at the first preset interval.

[0023] Optionally, after adjusting the frame rate of the frame extraction according to the recognition time consumption and the CPU occupancy rate, the method further includes:

[0024] Determining whether the adjusted frame rate is greater than a preset minimum frame rate;

[0025] When the judgment result is no, the number of target recognitions of the first algorithm is reduced according to preset rules.

[0026] Optionally, obtaining the recognition time consumed by the first algorithm for the extracted current video frame includes:

[0027] Acquire a first input time of the current video frame into the first algorithm, and a first output time of the recognition result of the first algorithm outputting the first algorithm;

[0028] The difference between the first input time and the first output time is determined as the identification time.

[0029] Optionally, obtaining the time consumed by the second algorithm for processing the recognition result of the first algorithm includes:

[0030] Acquire a second input time when the recognition result is input into the second algorithm, and a second output time when the processing result of the second algorithm is output from the second algorithm;

[0031] A difference between the second input time and the second output time is determined as the processing time.

[0032] According to another embodiment of the present invention, a frame rate adjustment device is provided, comprising:

[0033] A first acquisition module is used to obtain the recognition time consumed by the first algorithm for the extracted current video frame;

[0034] The second acquisition module is used to obtain the CPU occupancy rate of the central processing unit;

[0035] The adjustment module is used to adjust the frame rate of the frame extraction according to the recognition time consumption and the CPU occupancy rate.

[0036] Optionally, the device further comprises: a third acquisition module, configured to acquire the processing time consumed by the second algorithm for processing the recognition result of the first algorithm after acquiring the recognition time consumed by the first algorithm for the extracted current video frame;

[0037] Correspondingly, the adjustment module is also used to adjust the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption and the CPU occupancy rate.

[0038] Optionally, the adjustment module includes:

[0039] A reduction submodule, configured to reduce the frame rate at a first predetermined interval if the recognition time is greater than a ratio of a preset first minimum computing power threshold to the frame rate or the processing time is greater than a ratio of a preset second minimum computing power threshold to the frame rate;

[0040] The adjustment submodule is used to adjust the frame rate according to the recognition time, the processing time, the a core occupancy rate and the b core occupancy rate if the recognition time is less than the ratio of the preset first maximum computing power threshold to the frame rate or the processing time is greater than the ratio of the preset second maximum computing power threshold to the frame rate, wherein the CPU occupancy rate includes the a core occupancy rate and the b core occupancy rate.

[0041] Optionally, the regulating submodule includes:

[0042] A first increasing unit, configured to increase the frame rate at a second predetermined interval if the current occupancy rate of the a core is less than a preset a core idle threshold and the current occupancy rate of the b core is less than a preset b core idle threshold;

[0043] a second increasing unit, configured to increase the frame rate at the second predetermined interval if the current occupancy rate of the a core is less than the a core idle threshold and the current occupancy rate of the b core is not less than the b core idle threshold, or if the current occupancy rate of the a core is not less than the a core idle threshold and the current occupancy rate of the b core is less than the b core idle threshold, and the first algorithm and the second algorithm are both in the a core of the CPU, and the current occupancy rate of the a core is less than the a core idle threshold;

[0044] The adjustment unit is used to obtain the CPU occupancy rate of the first algorithm and the CPU occupancy rate of the second algorithm if the current occupancy rate of core a is greater than the preset busy threshold of core a or the current occupancy rate of core b is greater than the preset busy threshold of core b, and adjust the frame rate of the frame extraction according to the CPU occupancy rate of the first algorithm and the CPU occupancy rate of the second algorithm.

[0045] Optionally, the adjustment unit is also used to

[0046] If the first algorithm is in the a core of the CPU and the second algorithm is in the b core of the CPU, when the CPU occupancy rate of the first algorithm is greater than the a core idle threshold or the CPU occupancy rate of the second algorithm is greater than the b core idle threshold, the frame rate of the frame extraction is reduced at the first preset interval;

[0047] If the first algorithm and the second algorithm are both in the a core of the CPU, when the sum of the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm is greater than the a core busy threshold, the frame rate is reduced at the first preset interval.

[0048] Optionally, the device further comprises:

[0049] A judging module, used for judging whether the adjusted frame rate of the frame extraction is greater than a preset minimum frame rate;

[0050] The reduction module is used to reduce the number of target recognitions of the first algorithm according to preset rules when the judgment result is no.

[0051] Optionally, the first acquisition module includes:

[0052] A first acquisition submodule, used for acquiring a first input time of the current video frame into the first algorithm, and a first output time of the recognition result of the first algorithm outputting the first algorithm;

[0053] The first determining submodule is configured to determine a difference between the first input time and the first output time as the identification time consumption.

[0054] Optionally, the third acquisition module includes:

[0055] A second acquisition submodule, used to acquire a second input time of the recognition result being input into the second algorithm, and a second output time of the processing result of the second algorithm being output from the second algorithm;

[0056] The second determining submodule is used to determine the difference between the second input time and the second output time as the processing time.

[0057] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0058] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0059] Through the present invention, the recognition time consumed by the first algorithm for the extracted current video frame is obtained; the CPU occupancy rate of the central processing unit (CPU) is obtained; and the frame rate of the frame extraction is adjusted according to the recognition time consumption and the CPU occupancy rate. This can solve the problem that the frame rate processed by the fixed algorithm in the related technology cannot be applied to different scenarios and has poor flexibility. In scenarios with fewer targets (low pressure), the number of sent frames is increased to improve the algorithm effect. In scenarios with more targets (high pressure), the number of sent frames is reduced to ensure smooth pictures. This method can be flexibly applied to different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0061] Figure 1 1 is a hardware structure block diagram of a mobile terminal of the frame extraction frame rate adjustment method according to an embodiment of the present invention;

[0062] Figure 2 is a flow chart of a frame rate adjustment method according to an embodiment of the present invention;

[0063] Figure 3 Schematic diagram of frame extraction according to an embodiment of the present invention Figure 1 ;

[0064] Figure 4 Schematic diagram of frame extraction according to an embodiment of the present invention Figure 2 ;

[0065] Figure 5 is a schematic diagram of two-dimensional adjustment according to an embodiment of the present invention;

[0066] Figure 6 is a schematic diagram of a regulation model with different cores of an identification algorithm and a business algorithm according to an embodiment of the present invention;

[0067] Figure 7 is a schematic diagram of the adjustment of different cores of the identification algorithm and the business algorithm according to an embodiment of the present invention;

[0068] Figure 8 is a schematic diagram of a regulation model having the same core as the identification algorithm and the business algorithm according to an embodiment of the present invention;

[0069] Fig. 9 is a schematic diagram of adjustment of the same core of the recognition algorithm and the business algorithm according to an embodiment of the present invention;

[0070] Fig.10 is a flow chart of a frame extraction method according to an embodiment of the present invention;

[0071] Fig.11 is a schematic diagram of the regulation model with only the recognition algorithm;

[0072] Fig.12 This is a schematic diagram of the adjustment of the recognition algorithm only. Figure 1 ;

[0073] Fig.13 This is a schematic diagram of the adjustment of the recognition algorithm only. Figure 2 ;

[0074] Fig.14 4 is a block diagram of a frame rate adjustment device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0076] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0077] Example 1

[0078] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of the frame extraction frame rate adjustment method according to an embodiment of the present invention, such as Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0079] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the frame rate adjustment method of the frame extraction in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0081] In this embodiment, a frame rate adjustment method for frame extraction running on the above mobile terminal or network architecture is provided. Figure 2 is a flow chart of a frame rate adjustment method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0082] Step S202, obtaining the time consumed by the first algorithm to recognize the extracted current video frame;

[0083] In an embodiment of the present invention, the above-mentioned current video frame is extracted from the video based on the current frame rate, and the above-mentioned step S202 may specifically include: obtaining the first input time of the current video frame input into the first algorithm, and the first output time of the recognition result of the first algorithm outputting the first algorithm; and determining the difference between the first input time and the first output time as the recognition time.

[0084] In the embodiment of the present invention, the first algorithm may be an identification algorithm, and the second algorithm may be a service algorithm.

[0085] Step S204, obtaining the CPU occupancy rate of the central processing unit;

[0086] Step S206: adjusting the frame rate of frame extraction according to the recognition time consumption and the CPU occupancy rate.

[0087] Through the above steps S202 to S206, the recognition time of the first algorithm on the extracted current video frame is obtained; the CPU occupancy rate of the central processing unit is obtained; the frame rate of the frame extraction is adjusted according to the recognition time and the CPU occupancy rate. This can solve the problem that the frame rate processed by the fixed algorithm in the related technology cannot be applied to different scenarios and has poor flexibility. In the scenario with fewer targets (low pressure), the number of sent frames is increased to improve the algorithm effect. In the scenario with a large number of targets (high pressure), the number of sent frames is reduced to ensure smooth pictures. It can be flexibly applied to different scenarios.

[0088] In an embodiment of the present invention, after obtaining the recognition time of the first algorithm for the extracted current video frame, the processing time of the second algorithm for processing the recognition result of the first algorithm is obtained, and further, the second input time of the recognition result input into the second algorithm and the second output time of the processing result of the second algorithm are obtained; the difference between the second input time and the second output time is determined as the processing time; correspondingly, the above step S206 may specifically include: adjusting the frame rate of the frame extraction according to the recognition time, the processing time and the CPU occupancy rate. Among them, the first algorithm can be a recognition algorithm, and the second algorithm can be a business algorithm.

[0089] Further, adjusting the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption and the CPU occupancy rate may specifically include:

[0090] If the recognition time is greater than the ratio of the preset first minimum computing power threshold to the frame rate or the processing time is greater than the ratio of the preset second minimum computing power threshold to the frame rate, that is, t1>λ1 / F cur Or t2>θ1 / F cur , reducing the frame rate of the frame extraction at a first predetermined interval, wherein λ1 is the first minimum computing power threshold (computing power insufficient threshold), θ1 is the second minimum computing power threshold (computing power insufficient threshold), F cur is the frame rate of the frame extraction, t1 is the recognition time, and t2 is the processing time;

[0091] If the recognition time is less than the ratio of the preset first maximum computing power threshold to the frame rate or the processing time is greater than the ratio of the preset second maximum computing power threshold to the frame rate, that is, t1<λ0 / F cur And t2<θ0 / F cur, the frame rate of frame extraction is adjusted according to the recognition time consumption, the processing time consumption, the a core occupancy rate and the b core occupancy rate, wherein the CPU occupancy rate includes the a core occupancy rate and the b core occupancy rate, the first computing power minimum threshold is less than the first computing power maximum threshold rate, the second computing power minimum threshold rate is less than the second computing power maximum threshold rate, λ0 is the first computing power maximum threshold rate (computing power excess threshold rate), and θ0 is the second computing power maximum threshold rate (computing power excess threshold rate).

[0092] In the embodiment of the present invention, adjusting the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption, the a-core occupancy rate, and the b-core occupancy rate may further include:

[0093] If the current occupancy rate of core a is less than the preset idle threshold of core a and the current occupancy rate of core b is less than the preset idle threshold of core b, that is, core a cur <a0 and b cur <b0, increasing the frame rate at a second predetermined interval, wherein a0 is the idle threshold of the a core, and b0 is the idle threshold of the b core;

[0094] If the current occupancy rate of core a is less than the idle threshold of core a and the current occupancy rate of core b is not less than (greater than or equal to) the idle threshold of core b, that is, core a cur <a0 and b cur ≥ b0, or, the current occupancy rate of the a core is not less than (greater than or equal to) the a core idle threshold and the current occupancy rate of the b core is less than the b core idle threshold, that is, a cur ≥a0 and b cur <b0, and the identification algorithm and the business algorithm are both in the a core of the CPU, and the current occupancy rate of the a core is less than the a core idle threshold, that is, a cur <a0, increasing the frame rate at the second predetermined interval, and giving up adjusting the frame rate when the current occupancy rate of the a core is greater than or equal to the a core idle threshold; if the recognition algorithm and the business algorithm are in different cores of the CPU, giving up adjusting the frame rate;

[0095] If the current occupancy rate of core a is greater than the preset busy threshold of core a or the current occupancy rate of core b is greater than the preset busy threshold of core b, that is, core a cur > a1 or b cur> b1, obtain the CPU occupancy rate of the recognition algorithm and the CPU occupancy rate of the business algorithm, and adjust the frame rate according to the CPU occupancy rate of the recognition algorithm and the CPU occupancy rate of the business algorithm, wherein a1 is the busy threshold of the a core of the CPU, and b1 is the busy threshold of the b core of the CPU. Further, if the recognition algorithm is in the a core of the CPU and the business algorithm is in the b core of the CPU, when the CPU occupancy rate of the recognition algorithm is greater than the idle threshold of the a core (i.e., C alg >a0) or the CPU occupancy rate of the business algorithm is greater than the b-core idle threshold (i.e., C bu >b0), reduce the frame rate of the frame extraction at the first preset interval; if the CPU occupancy rate of the recognition algorithm is less than the idle threshold of the a core and the CPU occupancy rate of the business algorithm is less than the idle threshold of the b core C alg <a0 and C bu <b0, give up adjusting the frame rate of the frame extraction; if the recognition algorithm and the business algorithm are both in the a core of the CPU, when the sum of the CPU occupancy rate of the recognition algorithm and the CPU occupancy rate of the business algorithm is greater than the busy threshold of the a core, that is, C alg +C bu >a1, reduce the frame rate at the first preset interval, when the sum of the CPU occupancy rate of the recognition algorithm and the CPU occupancy rate of the business algorithm is less than the a core busy threshold and the current occupancy rate of the a core is greater than the a core busy threshold C alg +C bu <a0 and a cur >a1, give up adjusting the frame rate of the frame extraction.

[0096] In an optional embodiment, after adjusting the frame rate according to the recognition time and the CPU occupancy rate, it is determined whether the adjusted frame rate is greater than a preset minimum frame rate; if the judgment result is no, the number of target recognitions of the recognition algorithm is reduced according to preset rules.

[0097] Assume that the current device is P system 25 frame rate (F), the current algorithm frame rate is 10 frames (F cur ), that is, data is sent to the algorithm every 1 / 10s=100ms, and the algorithm unit has 100ms to perform calculation and analysis. If the analysis result is given within 100ms (such as t1=40ms), it will not affect the next frame sending. Otherwise, it will cause some problems: such as frame accumulation (when there is cache) and subsequent business process blocking, resulting in memory and CPU increases.

[0098] Figure 3 Schematic diagram of frame extraction according to an embodiment of the present invention Figure 1 ,like Figure 3As shown in the figure, the algorithm time consumption in the 4th frame is at the critical point, and performance problems begin to occur from the 5th frame. Factors that cause the increase in resource consumption (up and down fluctuations) of the recognition algorithm include: scene changes (such as background changes: camera pan rotation, lens zoom depth of field changes, lighting changes, etc.), changes in the number of detection targets (crowd gathering, traffic congestion, etc.), changes in target motion state (fast movement), and changes in resolution (switching video resolution). Figure 4 Schematic diagram of frame extraction according to an embodiment of the present invention Figure 2 ,like Figure 4 As shown, the load is usually balanced by binding algorithms to different CPU cores (multi-core case). However, as the number of algorithms accumulates, the CPU load may remain high. Memory consumption is similar.

[0099] After the accumulation begins, the performance impact may not be reflected in the CPU usage and memory size. For example, the recognition algorithm is running on dedicated hardware (NN IE, GPU, etc.), or the CPU itself is relatively strong, but the preset constant frame extraction value does not take this into account. At this time, the processing result returns slowly, which will cause the trajectory box and OSD attribute box of the real-time preview screen to lag (the video playback may be normal). For example, the current preview screen shows the 7th frame, but the intelligent result shows the 5th and 6th frames. If the target position changes slightly, the user may not notice it easily, but for large changes in the target position, the intelligent result display will be misplaced, or the tracking speed will not keep up with the target movement speed.

[0100] On the other hand, if the CPU usage is too high, it may not be due to the performance overhead of the algorithm. It may be occupied by other services. In this case, it is not necessary to actively give up resources.

[0101] Usually, the resource consumption of the recognition algorithm will directly affect the resource consumption of the business algorithm, and both of them will indirectly affect the upper-layer business (such as web, streaming media, communication with the backend, etc.).

[0102] For example, as the number of "motor vehicle" targets in the captured image increases, the recognition algorithm will consume more resources to identify all targets in the image (including various attributes) and output more results to the business algorithm, which will lead to increased resource consumption of the business algorithm (attribute association, license plate classification, database search, driver identity recognition, and screenshot optimization), which may affect higher-level application businesses (more track boxes, attribute display, violation and license plate duplicate detection, back-end communication volume, etc.).

[0103] Minimum algorithm frame rate requirements for different scenarios:

[0104] For scenes with little motion change, such as parking lots, underground garages, and indoors, the minimum frame rate sent to the algorithm can be relatively low, generally 5 to 8 frames.

[0105] For scenes with normal motion changes, such as residential areas, community streets, and office parks, the minimum frame rate sent to the algorithm is generally 8 to 10 frames.

[0106] For scenes with large motion changes, such as traffic and campus, the minimum frame rate sent to the algorithm is generally more than 10 frames.

[0107] The upper limit of the algorithm frame rate is generally half of the video frame rate, that is, F / 2. For example, if the P system is 25 frames, it is 12.5. This is a comprehensive empirical value based on the effect and hardware performance. But it is not absolute. For example, in scenes such as high-altitude dropping and highways, the higher the algorithm frame rate, the better. The minimum algorithm frame rate is not less than 12 frames.

[0108] Regarding the setting of the algorithm frame rate, the original practice is to evaluate the performance according to the predetermined scene and set a fixed value slightly higher than the minimum algorithm frame rate (i.e., a constant frame rate). For example, the general underground garage is 8 frames, the park is 12 frames, and the high-altitude throwing object is 20 frames.

[0109] The embodiments of the present invention dynamically adjust the frame rate and dynamically adjust the rate according to a strategy within a certain range, rather than extracting frames at a constant rate. In addition to frame extraction, through adjustment in two dimensions, there is another adjustment dimension for adjusting performance parameters. This can ensure smooth and stable images under stress scenarios, and improve the algorithm effect under general scenarios. It avoids adjustment according to fixed scenarios and improves the flexibility to cope with changing scenarios.

[0110] Figure 5 is a schematic diagram of two-dimensional adjustment according to an embodiment of the present invention, such as Figure 5 As shown in the figure, within a certain range, the algorithm frame rate is dynamically adjusted. The basis is that there is room for adjustment of the minimum algorithm frame rate and the algorithm frame rate of the performance upper limit. The basis for adjusting the recognition algorithm performance indicators is that performance indicators (such as the number of target recognition) will be distinguished by special cases. For example, in special cases, even the smoothness of the picture cannot be guaranteed, and the recognition rate cannot be guaranteed, so trade-offs must be made; in addition, the indicators may be reserved internally (the number of target recognition is slightly higher than the declared value, such as the actual maximum detectable 37 or 38 faces, and 36 are declared externally). The targets obtained by the recognition algorithm have confidence. The business algorithm will filter according to the business application situation at a confidence level higher than a certain confidence level, that is, the number of targets obtained by the recognition algorithm will be higher than the number of targets output by the business algorithm. This item is a cautious adjustment item because it affects the parameters. Among them, the algorithm frame rate adjustment can cope with changes in scenes, changes in the number of targets, changes in the motion state of targets, changes in resolution, etc.; while the algorithm indicator adjustment only copes with changes in the number of targets.

[0111] It should be noted that the recognition algorithm and the business algorithm do not necessarily run on different cores. Figure 6 is a schematic diagram of a regulation model with different cores of an identification algorithm and a business algorithm according to an embodiment of the present invention, such as Figure 6As shown, the recognition algorithm and the business algorithm are located in different cores, the recognition algorithm is located in core a, and the business algorithm is located in core b. Figure 7 is a schematic diagram of the adjustment of different cores of the recognition algorithm and the business algorithm according to an embodiment of the present invention, such as Figure 7 As shown, the frame extraction module extracts a video frame and sends it to the recognition algorithm module. After processing, the processing time of the recognition algorithm module is obtained, and its result is sent to the business algorithm module. After processing, the processing time of the business algorithm is obtained, and the two time consumption results are returned to the frame extraction module. Finally, the processing result is sent to the application business module. The frame extraction module adjusts the algorithm frame rate or algorithm index according to the two algorithm time consumptions and the two CPU occupancy rates. The two can also run on the same core. Figure 8 is a schematic diagram of a regulation model having the same core as the recognition algorithm and the business algorithm according to an embodiment of the present invention, such as Figure 8 As shown in the figure, the recognition algorithm and the business algorithm are located on the same core, and both the recognition algorithm and the business algorithm are located on core a. Several business threads are on the same core, and this binding allocation depends on the scenario, that is, the importance of various algorithms and businesses. Fig. 9 is a schematic diagram of the adjustment of the same core of the recognition algorithm and the business algorithm according to an embodiment of the present invention, such as Fig. 9 As shown, the frame extraction module extracts a video frame and sends it to the recognition algorithm module. After processing, the processing time of the recognition algorithm module is obtained, and the result is sent to the business algorithm module. After processing, the business algorithm processing time is obtained, and the two time consumption results are returned to the frame extraction module. Finally, the processing result is sent to the application business module. The frame extraction module adjusts the algorithm frame rate or algorithm index according to the two algorithm time consumptions and the two CPU occupancy rates.

[0112] When making adjustments, it is necessary to comprehensively consider scene changes (mainly target changes), algorithm time consumption, and CPU occupancy to make adjustments in two dimensions. Fig.10 is a flow chart of a frame extraction method according to an embodiment of the present invention. Fig.10 As shown, the adjustment method process is as follows:

[0113] Step S1001, initial configuration: set frame rate F, minimum algorithm frame rate F min , the highest algorithm frame rate F max , in general, F max = F / 2, usually not higher than this value, the effect is not obvious, but there are exceptions when the scene changes frequently, the frame rate cannot be lower than F min , otherwise the effect will be affected.

[0114] Step S1002, according to F cur Extract frames, the current algorithm frame rate is F cur , at the beginning, press the preset value (F min ~F max A value betweenmin Extract frames, that is, F cur =F min , maximum frame interval t0 = 1 / F min , current frame interval t cur =1 / F cur .

[0115] Step S1003, the recognition algorithm recognizes the current frame, calculates the recognition time t1, and the frame data (i.e. the current frame) is sent to the recognition algorithm. The time from the start of the input to the end of obtaining the analysis result is the recognition algorithm time (i.e. the above recognition time). At the same time, the target number Objs of the algorithm output results is recorded.

[0116] Step S1004, send the recognition result of the recognition algorithm to the business algorithm, and record the time consumed by the business algorithm (corresponding to the above-mentioned processing time). The business algorithm is not necessary and is related to the specific business scenario. If the subsequent processing is concentrated in the back end (NVR or service platform, etc.), it is only necessary to pass the algorithm result to the upper-level application, and the application will do a basic preview business display and send the result to the back end. If it is concentrated in the front end, such as the comparison of the access control face database data, the comparison of the vehicle position and the parking space of the parking space, and the billing system corresponding to the license plate recognition, it involves the business algorithm and "business application logic". The two concepts are similar. The difference is that in addition to the business logic, the former is more about data calculation, and the latter is more about business logic. However, it is more obvious in the design division of the camera architecture module, and they belong to different modules. It is precisely because of the different application scenarios that the selection of the aforementioned θ value will be different. The back-end processing does not even need to consider θ, and the front-end processing is considered as needed. When the complexity of the business exceeds the target recognition algorithm, the importance of θ exceeds λ, which is explained in the previous coefficient description section.

[0117] Step S1005, obtaining the CPU core a and core b occupancy rate a cur ,b cur , the CPU usage of the recognition algorithm C alg , CPU usage of business algorithm C bu . Query the CPU usage of the thread where the recognition algorithm is located C alg , the CPU occupancy rate of the thread where the business algorithm is located C bu The purpose is to examine whether the high CPU usage is caused by the algorithm. When creating a thread, the operating system allows the thread to be bound to a CPU core. After binding, it will always run on this core. If not specified, it will be automatically assigned by the operating system. In addition, the thread CPU usage is only for the thread itself. The occupancy of all threads on a core is the total CPU usage of the core. And usually a thread specializes in a certain function. The CPU core is public, and the thread is dedicated. The thread ID can be known when the thread is created, and the thread can be known on which core it runs after binding the core. This is the basis for the subsequent judgment.

[0118] Step S1006, according to the CPU occupancy rate and algorithm time-consuming technology current frame rate F cur , then execute step S1007 and step S1010;

[0119] Step S1007, determine F cur =F min whether the frame rate reduction condition is still met, if the result of the judgment is yes, executing step S1008, otherwise executing step S1009;

[0120] Specifically, if condition 1 is met, if F cur =F min , recovery algorithm index, where condition 1 is (t1<λ0 / F cur )﹠﹠(t2<θ0 / F cur )﹠﹠(a cur <a0)﹠﹠(b cur ≥b0);

[0121] F cur ++(<=F max ), that is, less than or equal to F max Range F cur Increase frame by frame;

[0122] When condition 2 is met, if F cur =F min , reduce the algorithm index, where condition 2 is (t1>λ1 / F cur )||(t2>θ1 / F cur )∣∣(a alg >a0﹠﹠a cur >a1)||(b bu >b0﹠﹠b cur >b1).

[0123] Step S1008, reducing algorithm indicators;

[0124] Step S1009, restore algorithm indicators;

[0125] Step S1010, completing the current frame processing.

[0126] Among them, when t1>λ1 / F cur (Don’t worry about t2, because the subsequent processing has been delayed at this time, which may affect the display to customers. The cache only reduces data loss and ensures normal storage of intelligent results). This means that the recognition algorithm takes a long time to process and will not be able to process the next frame in the future. The frame rate needs to be appropriately reduced. F cur Reduce frame by frame (until F min );

[0127] When t2>θ1 / F cur (t1 is not important, for the same reason as above. The values ​​of λ and θ are chosen to cooperate with the cache and take into account the effect). This means that the processing of the business algorithm takes a long time, and the next frame result output by the recognition algorithm will be backlogged (cached). In order to prevent the cache from being full and losing frames (processing results), the frame rate needs to be appropriately reduced. F cur Reduce frame by frame (until F min );

[0128] When t1<λ0 / F cur , and t2<θ0 / F cur , indicating that both the recognition algorithm and the business algorithm are relatively fast. cur <a0 and b cur <b0, that is, both cores are relatively idle, indicating sufficient computing power, F cur Increase frame by frame (until F max ); if a cur <a0 and b cur When a ≥ b0 cur ≥a0 and b cur < b0, depending on the binding of the algorithm and business threads: if the recognition algorithm and the business algorithm are on the same core (such as core a), and other businesses are on other cores (such as core b), a cur When <a0, F cur Increase frame by frame (until F max ), otherwise no adjustment; if the recognition algorithm and business algorithm are bound to different cores, that is, the business and algorithms are mixed on different cores, no adjustment is made, because the recognition algorithm is the source and will affect all subsequent modules.

[0129] when a cur > a1 or b cur > b1, if the recognition algorithm and business algorithm are bound to different cores (assuming they are bound to a and b respectively), and C alg >a0 or C bu >b0, indicating that it may be caused by the algorithm, F cur Reduce frame by frame (until F min ), if C alg <a0 and C bu <b0, do not adjust the frame rate;

[0130] If the recognition algorithm and the business algorithm are bound to the same core (assuming they are bound to a), when C alg +C bu >a1,F cur Reduce frame by frame (until F min ), when C alg +C bu <a0, and a cur>a1, indicating that the performance may be caused by other services, and the frame rate is not adjusted; the b core is not considered.

[0131] When F cur =F min , if the adjustment condition that causes the frame rate to decrease is still established (the condition is established, but the algorithm frame rate has been reduced to the bottom and is no longer reduced), adjust the "recognition algorithm performance index" to reduce the number of algorithm target recognition to the bottom line of the index. When the frame rate increase condition is triggered, first restore the target recognition number value, and then increase the frame rate frame by frame. In other cases except the above, maintain F cur No change. Press to get F cur The strategy is fed back to the algorithm frame rate adjustment module for frame extraction.

[0132] λ represents the threshold of the time-consuming adjustment parameter of the recognition algorithm. λ0 is the threshold of excess computing power, λ1 is the threshold of insufficient computing power, λ0<λ1, and λ0~λ1 is the transition interval. Because the general "frame extraction module", business algorithm, and recognition algorithm will have a cache (one frame to several frames), mainly to prevent some abnormal jumps (such as algorithm misdetection, business jams, etc.), this parameter is to prevent the feedback adjustment from being sensitive (also called "water level"), with a reference range of λ0 (0.5~0.7) and λ1 (0.8~1.0). The value is selected according to the number of caches and hardware resource configurations. For example: the algorithm frame rate varies from 8 to 12.5 frames, F min =8,F max =12.5, the corresponding algorithm processing time range is 80ms~125ms, the processing time within 50% can be considered as sufficient computing power, that is, 40ms~62.5ms, taking the extreme 62.5 / 80=0.78125, 40 / 125=0.32, that is, sufficient range: 0.32~0.78125; the processing time above 80% can be considered as insufficient computing power, that is, 64ms~100ms, taking the extreme 100 / 80=1.25, 64 / 125=0.512, that is, insufficient range: 0.512~1.25. Then select a suitable value within these two ranges.

[0133] θ represents the threshold of the time-consuming adjustment parameter of the business algorithm, θ0 is the threshold of excess computing power, θ1 is the threshold of insufficient computing power, θ0<θ1, and θ0~θ1 is the transition interval. In order to prevent the feedback adjustment from being too sensitive, the reference range θ0 is 0.6~0.8, and θ1 is 0.9~1.1. The derivation method is similar to λ.

[0134] Usually, the business algorithm is less sensitive to the algorithm frame rate than the recognition algorithm, so λ<θ can be set. Special scenarios may be the opposite, such as searching and comparing large amounts of database data, and accurately capturing images.

[0135] a, b represent the occupancy of CPU core a, b respectively (here we take dual core as an example, there are also 4 cores, the judgment is similar), a0, b0 are CPU idle thresholds, a1, b1 are CPU busy thresholds, a0<a1, b0<b1, a0~a1, b0~b1 are transition intervals to prevent feedback regulation from being too sensitive, the reference range is a0 (30%~50%), a1 (70%~90%), b0 (30%~50%), b1 (70%~90%), a cur , b cur They are the current CPU occupancy rates of cores a and b respectively. The selection of CPU occupancy rate is empirical and related to CPU main frequency and Catch performance. The higher the CPU frequency and the larger the Catch, the higher the busy value, and vice versa. Among them, for the selection of a0, a1, b0, and b1, the CPU consumption of the algorithm is generally higher than that of the business, so the CPU occupancy rate of the thread where the algorithm is located is always higher than that of the business. If the algorithm is located on the same core a, then generally a>b.

[0136] Fig.11 is a schematic diagram of the regulation model with only the recognition algorithm, such as Fig.11 As shown, there is only the recognition algorithm, which is in a different core from the application business, and the recognition algorithm is located in core a. Fig.12 This is a schematic diagram of the adjustment of the recognition algorithm only. Figure 1 ,like Fig.12 As shown, the frame extraction module extracts a video frame and sends it to the recognition algorithm module. After processing, the recognition algorithm module obtains the processing time, returns the time result to the frame extraction module, and finally sends the processing result to the application business module. The frame extraction module adjusts the algorithm frame rate or algorithm index according to the algorithm time and the two CPU occupancy rates.

[0137] Fig.13 This is a schematic diagram of the adjustment of the recognition algorithm only. Figure 2 ,like Fig.13 As shown, there is only the recognition algorithm, and it is in the same core as the application service, which is the most simplified case, equivalent to the single-core case. The frame extraction methods include:

[0138] 1. Initial configuration: set frame rate F, minimum algorithm frame rate F min , the highest algorithm frame rate F max , in general, F max = F / 2, usually not higher than this value, the effect is not obvious, but there are exceptions when the scene changes frequently, the frame rate cannot be lower than F min , otherwise it will affect the effect;

[0139] 2. According to F cur Extract frames, the current algorithm frame rate is F cur , at the beginning, press the preset value (F min ~F max A value betweenmin Extract frames, that is, F cur =F min , maximum frame interval t0 = 1 / F min , current frame interval t cur =1 / F cur .

[0140] 3. The frame data is sent to the recognition algorithm. The time taken by the recognition algorithm is counted from the time the frame data is sent to the time the analysis result is obtained. At the same time, the target number Objs of the algorithm output result is recorded;

[0141] 4. Query the CPU usage of the thread where the recognition algorithm is located C alg The purpose is to examine whether the high CPU usage is caused by the algorithm. When creating a thread, the operating system allows the thread to be bound to a CPU core. After binding, it will always run on this core. If not specified, it will be automatically assigned by the operating system. In addition, the thread CPU usage is only for the thread itself. The occupancy of all threads on a core is the total CPU usage of the core. And usually a thread specializes in a certain function. The CPU core is public, and the thread is dedicated. The thread ID can be known when the thread is created, and the thread can be known on which core it runs after binding the core. This is the basis for the subsequent judgment.

[0142] 5. Two-dimensional adjustment:

[0143] When t1>λ1 / F cur , indicating that the recognition algorithm takes a long time to process and cannot process the next frame in the future, so the frame rate needs to be appropriately reduced, F cur Reduce frame by frame (until F min );

[0144] When t1<λ0 / F cur , indicating that the recognition algorithm is faster. If the algorithm and application are on the same core (such as core a), it is the most simplified case and other cores are not considered. At this time, if a cur <a0, that is, the current core is relatively idle and has sufficient computing power, F cur Increase frame by frame (until F max ); if the algorithm (core a) and the application (core b) are heterogeneous, a cur <a0 and b cur <b0,F cur Increase frame by frame (until F max ), if a cur >a1, F cur Reduce frame by frame (until F min ), other situations are not adjusted.

[0145] If a cur >a1,F cur Reduce frame by frame (until F min), when C alg >a0, and a cur > a1, indicating that the performance may be caused by other services, and the frame rate is not adjusted;

[0146] When F cur =F min , if the adjustment condition for frame rate reduction is still met (the condition is met, but the algorithm frame rate has been reduced to the bottom and will not be reduced any more), adjust the recognition algorithm performance index, reduce the number of algorithm target recognition to the bottom line of the index, and when the frame rate increase condition is triggered, first restore the target recognition number value, and then increase the frame rate frame by frame;

[0147] In other cases, F-cur remains unchanged.

[0148] Press 5 for F cur The strategy is fed back to the algorithm frame rate adjustment module for frame extraction.

[0149] Example 2

[0150] According to another embodiment of the present invention, a frame rate adjustment device is provided. Fig.14 is a block diagram of a frame rate adjustment device according to an embodiment of the present invention. Fig.14 As shown, including:

[0151] A first acquisition module 142 is used to obtain the recognition time consumed by the first algorithm for the extracted current video frame;

[0152] The second acquisition module 144 is used to acquire the CPU occupancy rate of the central processing unit (CPU);

[0153] The adjustment module 146 is used to adjust the frame rate of the frame extraction according to the recognition time consumption and the CPU occupancy rate.

[0154] Optionally, the device further comprises: a third acquisition module, configured to acquire the processing time consumed by the second algorithm for processing the recognition result of the first algorithm after acquiring the recognition time consumed by the first algorithm for the extracted current video frame;

[0155] Correspondingly, the adjustment module is also used to adjust the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption and the CPU occupancy rate.

[0156] Optionally, the adjustment module includes:

[0157] A reduction submodule, configured to reduce the frame rate at a first predetermined interval if the recognition time is greater than a ratio of a preset first minimum computing power threshold to the frame rate or the processing time is greater than a ratio of a preset second minimum computing power threshold to the frame rate;

[0158] The adjustment submodule is used to adjust the frame rate according to the recognition time, the processing time, the a core occupancy rate and the b core occupancy rate if the recognition time is less than the ratio of the preset first maximum computing power threshold to the frame rate or the processing time is greater than the ratio of the preset second maximum computing power threshold to the frame rate, wherein the CPU occupancy rate includes the a core occupancy rate and the b core occupancy rate.

[0159] Optionally, the regulating submodule includes:

[0160] A first increasing unit, configured to increase the frame rate at a second predetermined interval if the current occupancy rate of the a core is less than a preset a core idle threshold and the current occupancy rate of the b core is less than a preset b core idle threshold;

[0161] a second increasing unit, configured to increase the frame rate at the second predetermined interval if the current occupancy rate of the a core is less than the a core idle threshold and the current occupancy rate of the b core is not less than the b core idle threshold, or if the current occupancy rate of the a core is not less than the a core idle threshold and the current occupancy rate of the b core is less than the b core idle threshold, and the first algorithm and the second algorithm are both in the a core of the CPU, and the current occupancy rate of the a core is less than the a core idle threshold;

[0162] The adjustment unit is used to obtain the CPU occupancy rate of the first algorithm and the CPU occupancy rate of the second algorithm if the current occupancy rate of core a is greater than the preset busy threshold of core a or the current occupancy rate of core b is greater than the preset busy threshold of core b, and adjust the frame rate of the frame extraction according to the CPU occupancy rate of the first algorithm and the CPU occupancy rate of the second algorithm.

[0163] Optionally, the adjustment unit is also used to

[0164] If the first algorithm is in the a core of the CPU and the second algorithm is in the b core of the CPU, when the CPU occupancy rate of the first algorithm is greater than the a core idle threshold or the CPU occupancy rate of the second algorithm is greater than the b core idle threshold, the frame rate of the frame extraction is reduced at the first preset interval;

[0165] If the first algorithm and the second algorithm are both in the a core of the CPU, when the sum of the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm is greater than the a core busy threshold, the frame rate is reduced at the first preset interval.

[0166] Optionally, the device further comprises:

[0167] A judging module, used for judging whether the adjusted frame rate of the frame extraction is greater than a preset minimum frame rate;

[0168] The reduction module is used to reduce the number of target recognitions of the first algorithm according to preset rules when the judgment result is no.

[0169] Optionally, the first acquisition module includes:

[0170] A first acquisition submodule, used for acquiring a first input time of the current video frame into the first algorithm, and a first output time of the recognition result of the first algorithm outputting the first algorithm;

[0171] The first determining submodule is configured to determine a difference between the first input time and the first output time as the identification time consumption.

[0172] Optionally, the third acquisition module includes:

[0173] A second acquisition submodule, used to acquire a second input time of the recognition result being input into the second algorithm, and a second output time of the processing result of the second algorithm being output from the second algorithm;

[0174] The second determining submodule is used to determine the difference between the second input time and the second output time as the processing time.

[0175] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0176] Example 3

[0177] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0178] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0179] S1, obtaining the recognition time consumed by the first algorithm for the extracted current video frame;

[0180] S2, obtain the CPU usage of the central processing unit;

[0181] S3, adjusting the frame rate of frame extraction according to the recognition time consumption and the CPU occupancy rate.

[0182] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0183] Example 4

[0184] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0185] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0186] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0187] S1, obtaining the recognition time consumed by the first algorithm for the extracted current video frame;

[0188] S2, obtain the CPU usage of the central processing unit;

[0189] S3, adjusting the frame rate of frame extraction according to the recognition time consumption and the CPU occupancy rate.

[0190] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0191] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0192] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A frame rate adjustment method, characterized in that: include: Obtaining the time consumed by the first algorithm to recognize the extracted current video frame; Get the CPU usage of the central processing unit; Adjusting the frame rate of frame extraction according to the recognition time consumption and the CPU occupancy rate; The method further includes: after obtaining the recognition time consumed by the first algorithm for the extracted current video frame, obtaining the processing time consumed by the second algorithm for processing the recognition result of the first algorithm; The frame rate of frame extraction is adjusted according to the recognition time, the processing time and the CPU occupancy rate, including: if the recognition time is greater than the ratio of a preset first minimum computing power threshold to the frame rate or the processing time is greater than the ratio of a preset second minimum computing power threshold to the frame rate, reducing the frame rate at a first predetermined interval; if the recognition time is less than the ratio of a preset first maximum computing power threshold to the frame rate or the processing time is greater than the ratio of a preset second maximum computing power threshold to the frame rate, adjusting the frame rate according to the recognition time, the processing time, the a core occupancy rate and the b core occupancy rate, wherein the CPU occupancy rate includes the a core occupancy rate and the b core occupancy rate.

2. The method according to claim 1, characterized in that Adjusting the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption, the a-core occupancy rate, and the b-core occupancy rate includes: If the current occupancy rate of the a-core is less than the preset a-core idle threshold and the current occupancy rate of the b-core is less than the preset b-core idle threshold, increasing the frame rate at a second predetermined interval; If the current occupancy rate of the a core is less than the a core idle threshold and the current occupancy rate of the b core is not less than the b core idle threshold, or the current occupancy rate of the a core is not less than the a core idle threshold and the current occupancy rate of the b core is less than the b core idle threshold, and both the first algorithm and the second algorithm are in the a core of the CPU, and the current occupancy rate of the a core is less than the a core idle threshold, increase the frame rate at the second predetermined interval; If the current occupancy of core a is greater than the preset busy threshold of core a or the current occupancy of core b is greater than the preset busy threshold of core b, obtain the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm, and adjust the frame rate of the frame extraction according to the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm.

3. The method according to claim 2, characterized in that Adjusting the frame rate of the frame extraction according to the CPU occupancy rate of the first algorithm and the CPU occupancy rate of the second algorithm includes: If the first algorithm is in the a core of the CPU and the second algorithm is in the b core of the CPU, when the CPU occupancy rate of the first algorithm is greater than the a core idle threshold or the CPU occupancy rate of the second algorithm is greater than the b core idle threshold, the frame rate of the extraction is reduced at a first preset interval; If the first algorithm and the second algorithm are both in the a core of the CPU, when the sum of the CPU occupancy of the first algorithm and the CPU occupancy of the second algorithm is greater than the a core busy threshold, the frame rate is reduced at the first preset interval.

4. The method according to claim 1, characterized in that: After adjusting the frame rate of the frame extraction according to the recognition time consumption and the CPU occupancy rate, the method further includes: Determining whether the adjusted frame rate is greater than a preset minimum frame rate; When the judgment result is no, the number of target recognitions of the first algorithm is reduced according to preset rules.

5. The method according to any one of claims 1 to 4, characterized in that Obtaining the recognition time of the first algorithm for the extracted current video frame includes: Acquire a first input time of the current video frame into the first algorithm, and a first output time of the recognition result of the first algorithm outputting the first algorithm; The difference between the first input time and the first output time is determined as the identification time.

6. The method according to any one of claims 1 to 4, characterized in that The time consumption of obtaining the second algorithm to process the recognition result of the first algorithm includes: Acquire a second input time when the recognition result is input into the second algorithm, and a second output time when the processing result of the second algorithm is output from the second algorithm; A difference between the second input time and the second output time is determined as the processing time.

7. A frame rate adjustment device, characterized in that: include: A first acquisition module is used to obtain the recognition time consumed by the first algorithm for the extracted current video frame; The second acquisition module is used to obtain the CPU occupancy rate of the central processing unit; An adjustment module, used for adjusting the frame rate of frame extraction according to the recognition time consumption and the CPU occupancy rate; The device further includes: a third acquisition module, configured to acquire the processing time consumed by the second algorithm for processing the recognition result of the first algorithm after acquiring the recognition time consumed by the first algorithm for the extracted current video frame; The adjustment module is further used to adjust the frame rate of the frame extraction according to the recognition time consumption, the processing time consumption and the CPU occupancy rate; Wherein, the adjustment module includes: A reduction submodule, configured to reduce the frame rate at a first predetermined interval if the recognition time is greater than a ratio of a preset first minimum computing power threshold to the frame rate or the processing time is greater than a ratio of a preset second minimum computing power threshold to the frame rate; The adjustment submodule is used to adjust the frame rate according to the recognition time, the processing time, the a core occupancy rate and the b core occupancy rate if the recognition time is less than the ratio of the preset first maximum computing power threshold to the frame rate or the processing time is greater than the ratio of the preset second maximum computing power threshold to the frame rate, wherein the CPU occupancy rate includes the a core occupancy rate and the b core occupancy rate.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images