A convolutional neural network inference optimization method based on video detection

Abstract

The application discloses a convolutional neural network inference optimization method based on video detection, comprising: a video frame counting module for recording the total amount of input video frames and the number of input video frames after updating background frames; a background frame storage module for storing background frames and inference results thereof and recording the interval duration of background frame updating; a similarity detection module for comparing the similarity of input video frames and background frames; a convolutional neural network inference calculation module for obtaining the inference result of input video frames; and a correlation matrix calculation module for calculating a threshold matrix and an accumulated frame matrix. The application is a low-cache-cost video task convolutional neural network inference optimization method, which does not need to consume a large amount of search time or learning cost, dynamically updates background frames, compares the similarity of background frames and input frames, reasonably reuses the inference result of background frames, reduces the calculation amount of convolutional neural networks in processing monitoring video, and shortens the inference time under acceptable accuracy loss.

Description

Technical Field

[0001] This invention belongs to the field of convolutional neural network algorithms. More specifically, it relates to a convolutional neural network inference optimization method based on video detection. Background Technology

[0002] Convolutional Neural Networks (CNNs) have become representative of the field of Deep Learning. During the development of CNNs, researchers have adopted more layers and more complex network structures to improve the accuracy of network models, leading to a continuous increase in the number of parameters and computational cost. Network models such as VGG16 and VGG19 even exceed 100 GFLOPs in computation. Currently, when using CNNs for video inference, each frame of the video is fed into the network model for inference calculations, directly impacting the processing efficiency of video inference tasks. In recent years, with the rapid development of the Internet of Things (IoT), deploying deep learning algorithms such as CNNs on resource-constrained IoT terminal devices to quickly respond to user requests and protect user privacy data has become a growing trend. Therefore, it is necessary to ensure that the loss of model accuracy is within an acceptable range while minimizing the computational cost of the CNN model, reducing memory usage, and shortening the model inference time.

[0003] When the video source is a real-time video surveillance system, the changes in pixels in the image sequence over time and the correlation between adjacent frames are significant. Specifically, consecutive frames fed into the network exhibit numerous similar scenes when there are no obviously moving objects, but when there is movement, the related pixel values ​​fluctuate considerably. This correlation is known as the spatiotemporal locality of video. Currently, some research utilizes the spatiotemporal locality of adjacent frames in video detection tasks to optimize the inference computation process of convolutional neural networks. There are three main approaches: one is to cache the computation results of each layer of key video frames, performing block-based searching for similar regions and reusing intermediate results for these regions; another also requires caching the computation results of each layer of key video frames, but only updates the computation results for the regions corresponding to the changed pixels based on the judgment of the video frame; and the third is to cache some key intermediate computation results, binding these intermediate results to the network output. If any intermediate computation result during inference is consistent with the previous one, the remaining computation is skipped and the result is output directly. All of the above solutions employ a space-for-time tradeoff. However, while optimizing the network, some add significant extra cache space for data storage, increasing lookup overhead. Others, while managing data using hash tables, also incur additional model caching and learning costs. These factors can create challenges and limitations when deployed on devices with limited resources. Summary of the Invention

[0004] 1. Technical problems to be solved

[0005] In most surveillance scenarios, the camera remains stationary for a significant portion of the time, with little movement of people or objects in the frame, while the background remains static or changes slowly. To address the shortcomings of existing technologies, this paper proposes a low-buffering-cost convolutional neural network (CNN) inference optimization scheme that leverages the slow background change characteristic of video frames. This reduces the computational load during CNN inference, increases its speed, and lowers memory consumption on resource-constrained devices.

[0006] 2. Technical Solution

[0007] This invention provides a convolutional neural network inference optimization method based on video detection. It aims to leverage the spatiotemporal locality between monitored video frames to construct background frames from long-term static regions within the video. The method caches the background frames and the final inference results processed by the convolutional neural network. A similarity comparison is performed between the current video frame and the background frame. When the similarity reaches a certain standard, the final inference result of the background frame is directly reused, reducing the computational load of the convolutional neural network in processing video data and shortening the video frame inference time with a slight loss of accuracy. The optimization diagram is shown below. Figure 1As shown, it includes the video frame counting module 101, the convolutional neural network inference calculation module 102, the similarity detection module 103, and the correlation matrix calculation module 104.

[0008] ■The video frame counting module 101 is used to record the total number of input video frames N and update the background frame. bg The number of subsequent input video frames is N′, with N initially set to 0 and N′ also initially set to 0 (background frames are not included in N′). When N = 1, the current input frame is the first video frame, frame1. The convolutional neural network inference calculation module 102 defaults to using the first video frame, frame1, as the initial background frame. The value of N is incremented by one for each input video frame. The background frame is then updated. bg After that, for each input video frame, the value of N′ is incremented by one. This increments by one when the background frame stored in the convolutional neural network inference calculation module 102... bg During the update, the value of N′ is set to 0;

[0009] ■ The convolutional neural network inference calculation module 102 is used to store the background frame. bg Calculate and store the background frame. bg Based on the inference results, record and update the background frame. bg Then until the next background frame update bg The time interval is determined by the similarity detection module 103. The value of the similarity marker variable β is read from the similarity detection module 103. If β = 0, the background frame stored in the convolutional neural network inference calculation module 104 is directly reused and output. bg Inference result. If β = 1, then the convolutional neural network inference calculation module 104 updates the background frame. bg Store the current input video frame. N background frame bg The updated background frame is obtained by running a convolutional neural network algorithm. bg The inference results are processed, new inference results are saved, previously saved inference results are deleted, Time is reset to 0, and the timing restarts. After each frame is processed, it is checked whether Time is greater than or equal to the set update duration t. If Time ≥ t, then the accumulated frames are calculated according to the correlation matrix calculation module 104.

[0010] Matrix A∈R m×n Calculate the historical average frame rate and update the background frame using the historical average frame rate.

[0011] ■ The similarity detection module 103 is used to compare the input video frames. N With background frame bgThe similarity is calculated by comparing the input video frames. N With background frame bg Subtracting the pixel values ​​at each corresponding pixel point yields a similarity matrix S∈R. m×n The threshold matrix T∈R calculated in the correlation matrix calculation module 104 is related to... m×n Perform a numerical comparison of each matrix element, counting the number of elements in the similarity matrix whose values ​​are greater than the corresponding element values ​​in the threshold matrix, denoted as num. Define a similarity threshold variable α (0 ≤ α ≤ m × n, where α is an integer) and a similarity marker variable β (β is initially set to 0 and reset to 0 before each similarity evaluation). When num ≤ α, the input video frame is considered to be... N With background frame bg The input video frame contains the same object, and the displacement of the object is very small. N With background frame bg If the similarity is very high, β = 0; otherwise, the input video frame is considered to be... N With background frame bg They are not similar, so set β = 1;

[0012] ■ The correlation matrix calculation module 104 is used to calculate the threshold matrix T∈R m×n and the accumulated frame matrix A∈R m×n (Each video frame has a size of m×n). To evaluate the input video frame... N With background frame bg The similarity between them depends on the background frame. bg to video frame N-1 The threshold matrix T is calculated by taking the average change in pixel value at each pixel point from the N′ historical video frames between them. The accumulated frame matrix A∈R m×n Based on video frames frame1 to frame N It is calculated by summing the pixel values ​​of each pixel point in the N historical video frames between them.

[0013] This invention provides a convolutional neural network inference optimization method based on video detection, the overall flowchart of which is shown below. Figure 2 As shown, the specific steps are as follows:

[0014] S1: Input the video frame, then proceed to step S2;

[0015] S2: The total number of video frames N input to the video frame counting module 101 is incremented by 1, and the current video frame is [frame number missing]. N Then proceed to step S3;

[0016] S3: Determine whether N is greater than 1. If N ≤ 1, go to step S4; if N > 1, go to step S10;

[0017] S4: The convolutional neural network inference calculation module 102 sets the video frame when N equals 1 as the initial background frame, and the video frame frame N as the background frame frame bg is saved, the inference result of frame bg is calculated and saved, Time starts timing, and then go to step S5;

[0018] S5: The correlation matrix calculation module 104 calculates the cumulative frame matrix A ∈ R m×n , and then go to step S6;

[0019] S6: The convolutional neural network inference calculation module 102 determines whether Time is greater than or equal to the set update duration t. If Time < t, go to step S7; if Time ≥ t, go to step S8

[0020] S7: Determine whether there is still a video frame input. If so, go to S1; otherwise, end;

[0021] S8: The convolutional neural network inference calculation module 102 reads the cumulative frame matrix A ∈ R in the correlation matrix calculation module 104 m×n calculates and sets the historical average frame as the background frame frame bg is saved, the inference result of frame bg is calculated and saved, Time is set to 0 and starts timing again, and then go to step S9;

[0022] S9: The correlation matrix calculation module 104 resets the elements in the threshold matrix T ∈ R m×n to 0, and then go to step S20;

[0023] S10: The video frame counting module 101 calculates N' plus 1, and then go to step S11;

[0024] S11: Determine whether N' is greater than 1. If N' ≤ 1, go to step S15; if N' > 1, go to step S12;

[0025] S12: The similarity detection module 103 calculates the similarity matrix S ∈ R between the input video frame frame N and the background frame frameelement by element, counts the number of elements greater than the threshold matrix as num, and then go to step S13; ​​​​​

[0026] S13: The similarity detection module 103 determines whether num is greater than the similarity threshold variable α. If num≤α, then proceed to step S14; if num>α, then proceed to step S17.

[0027] S14: The similarity detection module 103 sets the similarity marker variable β to 0, and then proceeds to step S15;

[0028] S15: The convolutional neural network inference calculation module 102 reads the frame bg The inference result is output directly, and then the process proceeds to step S16;

[0029] S16: The correlation matrix calculation module 104 calculates and updates the threshold matrix T∈R m×n and the accumulated frame matrix A∈R m ×n Then proceed to step S6;

[0030] S17: The similarity detection module 103 sets the similarity marker variable β to 1, and then proceeds to step S18;

[0031] S18: The convolutional neural network inference calculation module 102 sets the video frame. N As a background frame bg Save and calculate the frame. bg The inference result is saved, Time is set to 0 and the timing is restarted, and then the process proceeds to step S19;

[0032] S19: The correlation matrix calculation module 104 resets the threshold matrix T∈R m×n If the elements within are 0, calculate and update the accumulated frame matrix A∈R. m×n Then proceed to step S20;

[0033] S20: The video frame counting module 101 sets N' to 0, and then proceeds to step S6;

[0034] like Figure 2 The calculation process of this invention mainly includes three stages: initialization stage, similar frame processing stage, and background frame update stage. The initialization stage mainly includes steps S4 and S5, the similar frame processing stage mainly includes steps S14, S15, and S16, and the background frame update stage mainly includes steps S8, S9, S17, S18, S19, and S20.

[0035] 3. Beneficial effects

[0036] The advantages of this invention are: it does not require retraining the convolutional neural network model, does not require large-scale cache space, and does not consume a lot of search time or learning cost. By dynamically updating the background frame and comparing the similarity between the background frame and the current input frame, the inference result of the background frame is reused. This shortens the inference time of the convolutional neural network based on video tasks with acceptable accuracy loss. It is a low-caching-cost optimization method for convolutional neural network inference in video tasks.

[0037] Compared to CBinfer and DeepCache methods, this invention uses less cache space, eliminating the need for large-scale caching of repeatedly computed data. Furthermore, this invention provides more granular cached computation results, resulting in lower cost for similarity comparison when reusing computational results. Compared to the GATI method, this invention uses less cache space and eliminates the need to build a model to learn cache hit rates, thus avoiding additional learning costs to retain high-hit-rate results. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of a convolutional neural network inference optimization method based on video detection according to the present invention;

[0039] Figure 2 This is a flowchart of a convolutional neural network inference optimization method based on video detection according to the present invention;

[0040] Figure 3 The correlation matrix calculation module 104 of one embodiment of the present invention calculates the threshold matrix T∈R m×n Schematic diagram;

[0041] Figure 4 The correlation matrix calculation module 104 of one embodiment of the present invention updates the accumulated frame matrix A∈R. m×n Schematic diagram; Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the following description, in conjunction with the accompanying drawings and embodiments, provides a more detailed explanation of the invention. Based on these detailed descriptions, those skilled in the art will be able to clearly understand and implement this application. Without departing from the principles of this application, features in various embodiments can be combined to obtain new implementations, or certain features in some embodiments can be substituted to obtain other preferred implementations.

[0043] In this embodiment, the common video object detection model YOLO is used to process the video frame sequence captured by the surveillance camera. When the first video frame in the video frame sequence is input, the total number of video frames N=1 and the updated background frame are input into the video frame counting module 101.bg The number of subsequent input video frames N′ = 0. The convolutional neural network inference calculation module 102 defaults to storing the first input video frame1 as the background frame. bg The convolutional neural network inference module 102 uses YOLO to calculate the background frame. bg The inference result is stored in the convolutional neural network inference calculation module 102 as a background frame. bg The pixel value of each pixel is used to count the time from the start of the timer (using the variable Time) until the next background frame update. The threshold matrix T∈R in the correlation matrix calculation module 104... m×n and the accumulated frame matrix A∈R m×n Initially, all are zero-valued matrices. When N = 1, the threshold matrix T ∈ R. m×n Still a zero-value matrix, the accumulated frame matrix A∈R m×n Update to the sum of video frame1 and the initial accumulated frame matrix A.

[0044] When inputting the second video frame in the input video frame sequence, the total number of input video frames N=2 and the updated background frame frame... bg The number of subsequent input video frames is N′ = 1. The video frame with N′ = 1 is guaranteed to be the first frame after the background frame has been updated, and by default, it has a high similarity to the background frame, allowing direct reuse of the background frame's inference results. The convolutional neural network inference calculation module 102 reads the frame... bg The inference result is output directly. The correlation matrix calculation module 104 calculates the threshold matrix T∈R. m×n And update the accumulated frame matrix A∈R m×n Threshold matrix T∈R m×n The calculation method is as follows Figure 3 As shown, first calculate the difference between the video frame 2 and the background frame 1. bg For each corresponding pixel point, the difference matrix D∈R is obtained. m×n Then, each element of the difference matrix D is divided by N′ to obtain the threshold matrix T. The accumulated frame matrix A∈R m×n The update method is as follows Figure 4 As shown, video frame 2 is added to the accumulated frame matrix A to obtain the updated accumulated frame matrix A. After processing each video frame, the convolutional neural network inference calculation module 102 checks whether Time is greater than or equal to the set update duration. When Time ≥ t, the convolutional neural network inference calculation module 102 reads the accumulated frame matrix A∈R from the correlation matrix calculation module 104. m×n Divide by N′ to calculate the historical average frame, and set it as the new background frame. bgThen, the video frame counting module 101 resets N′ to 0, the convolutional neural network inference calculation module 102 resets Time to 0 and starts timing again, and the correlation matrix calculation module 104 resets the threshold matrix T∈R. m×n This is a zero-valued matrix. Each background frame update resets N′, Time, and the threshold matrix T∈R. m ×n .

[0045] When the third video frame in the input video frame sequence is input, the total number of input video frames N=3 and the updated background frame is... bg The number of subsequent input video frames N′ = 2. The similarity detection module 103 calculates the similarity between the input video frame3 and the background frame. bg The similarity matrix S∈R is obtained by performing matrix subtraction. m×n And with the threshold matrix T∈R in the correlation matrix calculation module 104 m×n Element-by-element comparison is performed, counting the number of elements (num) greater than the threshold matrix T. The value of num is then compared to the similarity threshold variable α. When num ≤ α, the similarity label variable β = 0; otherwise, β = 1. When β = 0, the input video frame frame3 and the background frame frame... bg The similarity is very high, and the convolutional neural network inference calculation module 102 directly reads the background frame. bg Inference results. When β = 1, the input video frame 3 and the background frame 4 are... bg They are not similar; the convolutional neural network inference calculation module 102 stores the input video frame 3 as a representation of the background frame 3. bg The convolutional neural network inference calculation module 102 calculates and stores the background frame. bg The convolutional neural network inference results, reset N′, Time and threshold matrix T∈R m×n .

[0046] Subsequent video frames in the input video frame sequence fall into two categories: one is video frames... N With background frame bg Similarly, the background frame is read directly from the convolutional neural network inference calculation module 102. bg Inference results; another type is video frames. N With background frame bg If they are dissimilar, a background frame update is required. The inference results for the background frame are recalculated, and N′, Time, and the threshold matrix T∈R are reset accordingly. m×nAfter processing each video frame, the convolutional neural network inference calculation module 102 must determine whether Time is greater than or equal to the set update duration to prevent deviations in the inference results caused by prolonged periods without updating the background frame.

[0047] Although this application has been described above with reference to specific embodiments, those skilled in the art will understand that many modifications can be made to the configurations and details disclosed in this application within the principles and scope of the disclosure. The scope of protection of this application is determined by the appended claims, and the claims are intended to cover all modifications included in the literal meaning or scope of equivalents of the technical features in the claims.

Claims

1. A convolutional neural network inference optimization method based on video detection, characterized in that: The method includes a video frame counting module, a convolutional neural network inference calculation module, a similarity detection module, and a correlation matrix calculation module; The video frame counting module is used to record the total number of input video frames and the number of input video frames after updating the background frame; The convolutional neural network inference calculation module is used to store background frames, calculate and store the inference results of background frames, and record the time between updating the background frame and the next update of the background frame. The similarity detection module is used to compare the similarity between the input video frame and the background frame; The correlation matrix calculation module is used to calculate the threshold matrix and the accumulated frame matrix; The method includes the following steps: S1: Input video frame Then proceed to step S2; S2: The total number of video frames N input to the video frame counting module is incremented by 1, and the current video frame is... Then proceed to step S3; S3: Determine if N is greater than 1.

1. Then proceed to step S4. If N 1. Then proceed to step S10; S4: The convolutional neural network inference calculation module sets the video frame when N equals 1 as the initial background frame, video frame As background frame Save, calculate The inference results are saved, Time starts timing, and then proceed to step S5; S5: The correlation matrix calculation module calculates the accumulated frame matrix. Then proceed to step S6; S6: The convolutional neural network inference calculation module determines whether Time is greater than or equal to the set update duration t. If Time... If t, then proceed to step S7. If Time If t, then proceed to step S8; S7: Determine if there are still video frames input. If yes, proceed to S1; otherwise, end. S8: The convolutional neural network inference calculation module reads the accumulated frame matrix from the correlation matrix calculation module. Calculate and set the historical average frame as the background frame. Save, calculate Save the inference result, set Time to 0 to restart the timer, and then proceed to step S9; S9: The correlation matrix calculation module resets the threshold matrix. The element in the array is 0, then proceed to step S20; S10: The video frame counting module calculates N' plus 1, and then proceeds to step S11; S11: Determine if N' is greater than 1.

1. Then proceed to step S15. If N' 1. Then proceed to step S12; S12: The similarity detection module calculates the input video frames. With background frame Similarity matrix , and threshold matrix Compare element by element, count the number of elements in the matrix that are greater than the threshold (num), and then proceed to step S13. S13: The similarity detection module determines whether num is greater than the similarity threshold variable α. If num If α, then proceed to step S14. If num Then proceed to step S17; S14: The similarity detection module sets similarity marker variables. for Then proceed to step S15; S15: The convolutional neural network inference calculation module reads... The inference result is output directly, and then the process proceeds to step S16; S16: The correlation matrix calculation module calculates and updates the threshold matrix. and accumulated frame matrix Then proceed to step S6; S17: The similarity detection module sets similarity marker variables. for Then proceed to step S18; S18: The convolutional neural network inference calculation module sets the video frame. As background frame Save, calculate The inference result is saved, Time is set to 0 and the timing is restarted, and then the process proceeds to step S19; S19: The correlation matrix calculation module resets the threshold matrix. If the elements inside are 0, calculate and update the accumulated frame matrix. Then proceed to step S20; S20: The video frame counting module sets N' to 0, and then proceeds to step S6; Specifically, the correlation matrix calculation module in S16 calculates the threshold matrix based on the average change in pixel value at each pixel point in all historical video frames between the background frame and the input video frame. Accumulated frame matrix It is calculated based on the sum of the pixel values ​​at each pixel point in all historical video frames.

2. The convolutional neural network inference optimization method based on video detection as described in claim 1, characterized in that: The video frame counting module records the total number of input video frames N and the number of input video frames N′ after the background frame is updated. The initial value of N is 0, the initial value of N′ is 0, and the background frame is not included in N′. The value of N is incremented by one for each input video frame. The value of N′ is incremented by one for each input video frame after the background frame is updated. When the background frame stored in the convolutional neural network inference calculation module is updated, the value of N′ is set to 0.

3. The convolutional neural network inference optimization method based on video detection as described in claim 1, characterized in that: The convolutional neural network inference calculation module stores the background frame, runs the convolutional neural network algorithm to calculate and store the inference result of the background frame, and directly reuses the stored background frame inference result for video frames similar to the background frame; it records the time between updating the background frame and the next updating the background frame to ensure that the background frame can be updated in a timely manner when the background frame is not updated for a long time.

4. The convolutional neural network inference optimization method based on video detection as described in claim 1, characterized in that: The similarity detection module subtracts the pixel values ​​of each corresponding pixel point in the input video frame from those in the background frame to obtain a similarity matrix, which is then compared with the threshold matrix of the correlation matrix calculation module. The similarity between the input video frame and the background frame is determined by comparing the values ​​of each matrix element.

5. The convolutional neural network inference optimization method based on video detection as described in claim 1, characterized in that: The calculation process comprises three stages: initialization, similar frame processing, and background frame update. The initialization stage includes steps S4 and S5; the similar frame processing stage includes steps S14, S15, and S16; and the background frame update stage includes steps S8, S9, S17, S18, S19, and S20. Each time the background frame is updated, the video frame counting module resets N′ to 0, the convolutional neural network inference calculation module resets Time to 0 and starts timing again, and the correlation matrix calculation module resets the threshold matrix. It is a zero-valued matrix.

6. The convolutional neural network inference optimization method based on video detection as described in claim 1, characterized in that: In step S8, the convolutional neural network inference calculation module reads the accumulated frame matrix from the correlation matrix calculation module. Divide by N′ to calculate the historical average frame, set the historical average frame as the background frame and save it, run the convolutional neural network algorithm to obtain the new background frame inference result and save it.

Images