Video processing method, device, network video recorder and storage medium

Abstract

The embodiment of the present invention provides a video processing method, device, network video recorder and storage medium, which relates to the field of video processing technology. The method dynamically adjusts the size of the first cache, the second cache and the third cache according to the network status and business load, and uses three caches to hierarchically store received video frames, to-be-processed video frames and to-be-displayed video frames, effectively avoiding resource contention and resource waste, and can optimize resource utilization, thereby improving the efficiency and reliability of video frame processing and improving the overall stability of the video processing system.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video processing method, device, network video recorder and storage medium. Background Art

[0002] With the rapid development of technologies such as high-definition video, image processing, and video surveillance, the amount of data that video processing systems need to process is becoming larger and larger. Users' requirements for the real-time and high-efficiency of video processing systems are also increasing. When purchasing video processing solutions, users usually have clear requirements for the system's processing performance to meet business needs. In many business scenarios, the performance of video processing is not just a matter of speed, but can even affect whether the business itself can run normally. Therefore, how to improve the efficiency of video processing has become a technical issue that cannot be underestimated. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a video processing method, device, network video recorder and storage medium, which can improve the efficiency and reliability of video frame processing.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, the present invention provides a video processing method, which is applied to a network video recorder, wherein a first cache, a second cache and a third cache are created on the network video recorder; the first cache is used to store received video frames, the second cache is used to store video frames to be processed, and the third cache is used to store video frames to be displayed; the method comprises:

[0006] Adjust the sizes of the first cache, the second cache, and the third cache according to network conditions and business load;

[0007] Moving video frames that meet the video processing condition from the first buffer to the second buffer;

[0008] Obtaining a to-be-processed video frame from the second buffer;

[0009] A video frame to be displayed is determined according to the video frame to be processed, and the video frame to be displayed is saved in the third buffer.

[0010] In an optional implementation manner, moving the video frames that meet the video processing condition from the first cache to the second cache includes:

[0011] Move video frames whose decoding timestamps are greater than or equal to the display timestamps from the first cache to the second cache;

[0012] Move video frames whose actual cache time exceeds a cache time threshold from the first cache to the second cache;

[0013] When the first buffer is empty, the received video frame is saved in the second buffer.

[0014] In an optional implementation manner, obtaining the to-be-processed video frame from the second cache includes:

[0015] The to-be-processed video frames are obtained from the second cache in order of priority from high to low; or, the to-be-processed video frames are obtained from the second cache in sequence.

[0016] In an optional implementation manner, determining the video frame to be displayed according to the video frame to be processed includes:

[0017] When the preset frame skipping switch is turned off, decoding the video frame to be processed to obtain the video frame to be displayed;

[0018] When the frame skipping switch is turned on, the video frames to be processed that meet the preset frame skipping interval are decoded to obtain the video frames to be displayed.

[0019] In an optional embodiment, the method further comprises:

[0020] When the frame supplementation condition is met, a transition frame is generated according to two adjacent video frames in the third cache, and the transition frame is saved in the third cache; the transition frame is used to smooth the motion trajectory between the two adjacent video frames.

[0021] In an optional implementation manner, the network condition includes bandwidth, delay, jitter, packet loss rate and throughput; the business load includes CPU utilization, memory utilization, bandwidth utilization, disk read and write speed and cache utilization; and adjusting the sizes of the first cache, the second cache and the third cache according to the network condition and the business load includes:

[0022] Determine a network status score according to the bandwidth, the delay, the jitter, the packet loss rate, and the throughput;

[0023] Determine a business load score according to the CPU utilization, the memory utilization, the bandwidth utilization, the disk read and write speed, and the cache utilization;

[0024] The sizes of the first cache, the second cache, and the third cache are updated according to the network status score and the service load score.

[0025] In an optional implementation manner, updating the sizes of the first cache, the second cache, and the third cache according to the network status score and the service load score includes:

[0026] Determine a comprehensive score based on the network status score, the network status weight, the business load score and the business load weight, and update the sizes of the first cache, the second cache and the third cache based on the comprehensive score; or, update the sizes of the first cache, the second cache and the third cache based on the worst score of the network status score and the business load score.

[0027] In a second aspect, the present invention provides a video processing device, which is applied to a network video recorder. A first cache, a second cache and a third cache are created on the network video recorder; the first cache is used to store received video frames, the second cache is used to store video frames to be processed, and the third cache is used to store video frames to be displayed; the device comprises:

[0028] An adjustment module, configured to adjust the sizes of the first cache, the second cache, and the third cache according to network conditions and service load;

[0029] The processing module is used to move the video frames that meet the video processing conditions from the first cache to the second cache; obtain the video frames to be processed from the second cache; determine the video frames to be displayed based on the video frames to be processed, and save the video frames to be displayed in the third cache.

[0030] In a third aspect, the present invention provides a network video recorder, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the video processing method described in any of the aforementioned implementations.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the video processing method as described in any one of the aforementioned embodiments.

[0032] Compared with the prior art, the video processing method, device, network video recorder and storage medium provided by the embodiments of the present invention dynamically adjust the sizes of the first cache, the second cache and the third cache according to the network status and business load, and use the three caches to hierarchically store the received video frames, the video frames to be processed and the video frames to be displayed, effectively avoiding resource contention and resource waste, and optimizing resource utilization, thereby improving the efficiency and reliability of video frame processing and improving the overall stability of the video processing system.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A schematic diagram of an application scenario for processing video frames in the prior art is shown.

[0036] Figure 2 A schematic diagram of another application scenario for processing video frames in the prior art is shown.

[0037] Figure 3 A schematic diagram of an application scenario of complex processing and simple processing of video frames in the prior art is shown.

[0038] Figure 4 A schematic flow chart of a video processing method provided by an embodiment of the present invention is shown.

[0039] Figure 5 Shows Figure 4 A flow chart of sub-steps of step S10 and step S20.

[0040] Figure 6 A block diagram of a video processing device provided by an embodiment of the present invention is shown.

[0041] Figure 7 A block diagram of a network video recorder provided by an embodiment of the present invention is shown.

[0042] Icon: 100 - network video recorder; 110 - memory; 120 - processor; 130 - communication module; 200 - video processing device; 201 - adjustment module; 202 - processing module. DETAILED DESCRIPTION

[0043] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0045] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0046] In the prior art, a video processing system usually uses a network video recorder (Network Video Recorder, NVR for short) to receive, process, record and store data from an Internet Protocol Camera (IPC for short) or a network video stream.

[0047] With the rapid growth of network videos, the amount of data that video processing systems need to process is increasing, and the efficiency of video processing seriously affects the normal operation of business. For example, in the field of video surveillance, real-time processing and analysis of video streams is an important part of ensuring security. In high-definition video production, efficient video processing is a key factor in ensuring high-quality output.

[0048] The inventor has found that, at present, the video processing system usually adopts a single-level buffer pool. The video processing system stores all video frames in a shared buffer pool to process the video frames. The specific processing steps are that the video source sends the video frame to the video processing system, the video processing system stores all the received video frames in a shared buffer pool, and multiple processing threads (such as processing thread 1, processing thread 2 and processing thread 3) sequentially obtain the video frame from the buffer pool for processing. When the processing thread finishes processing a video frame, it obtains a new video frame from the buffer pool to continue processing.

[0049] When the speed at which the video source sends video frames is faster than the speed at which the processing thread processes video frames, video frames will accumulate in the buffer pool. When the buffer pool is full, the video processing system will block the video input source and stop receiving new video frames until the video frames in the buffer pool are processed. Then the video processing system will continue to receive new video frames. Figure 1 shown.

[0050] In the prior art, since all video frames share a cache pool, multiple threads compete to access the same cache pool, which will lead to resource contention, thereby reducing the concurrent processing capability and response speed of the video processing system. In addition, due to load imbalance, some threads are overly busy, while other threads are idle, affecting the effective use of system resources. Since video frames are not processed according to time nodes, processing delays increase, system scalability is limited, and it is impossible to effectively cope with high concurrency and complex processing requirements.

[0051] by Figure 2 For example, suppose that the video processing system uses four processing threads to process video frames concurrently, and each processing thread needs to obtain video frames from a single-level cache pool for processing. First, processing thread 1 and processing thread 2 try to access the single-level storage pool at the same time, resulting in resource contention. Subsequently, processing thread 3 and processing thread 4 try to access at the same time, further exacerbating resource contention. This resource contention will cause the access speed of all processing threads to slow down and the processing delay to increase. Ultimately, the overall performance of the video processing system will decline, and video frames cannot be processed efficiently.

[0052] by Figure 3 For example, suppose the video processing system uses four processing threads to process video frames concurrently, where processing threads 1 and 2 perform complex processing on video frames, while processing threads 3 and 4 perform simple processing on video frames. Since all processing threads obtain video frames from the same single-level storage pool, the processing threads that perform complex processing on video frames will occupy more resources, while the processing threads that perform simple processing are idle. This load imbalance will also lead to a decrease in the performance of the video processing system. In buses or ordinary outdoor monitoring scenarios, external environmental factors or network delays further exacerbate the problems of decreased video frame processing capabilities and system freezes.

[0053] In order to solve the above problems, the core improvement of the present invention is to dynamically adjust the sizes of the first cache, the second cache and the third cache according to network conditions and business loads, and use the three caches to hierarchically store the received video frames, the video frames to be processed and the video frames to be displayed, so as to effectively avoid resource contention and resource waste, optimize resource utilization, thereby improving the efficiency and reliability of video frame processing, and improving the overall stability of the video processing system.

[0054] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] Please refer to Figure 4 , Figure 4A schematic flow chart of a video processing method provided by an embodiment of the present invention is shown. The method is applied to a network video recorder, and a first cache, a second cache, and a third cache are created on the network video recorder; the first cache is used to store received video frames, the second cache is used to store video frames to be processed, and the third cache is used to store video frames to be displayed; the method includes the following steps:

[0056] Step S10, adjusting the sizes of the first cache, the second cache, and the third cache according to the network status and the service load.

[0057] In the embodiment of the present invention, the network status and service load of the network video recorder are obtained in real time, and the size of the first cache, the size of the second cache and the size of the third cache are dynamically adjusted according to the obtained network status and service load, so as to effectively optimize resource utilization.

[0058] Step S20: moving video frames that meet the video processing condition from the first cache to the second cache.

[0059] In an embodiment of the present invention, the video processing system includes a network video recorder and a network camera. When the network video recorder successfully establishes a connection with the network camera and pulls a video stream, the network video recorder stores all received video frames in a first cache.

[0060] The video frames that meet the video processing conditions are gradually moved from the first cache to the second cache, and the video frames that do not meet the video processing conditions at the current moment continue to be stored in the first cache, thereby ensuring that the video frames that meet the video processing conditions are processed first, effectively improving the processing efficiency of the video frames that meet the video processing conditions.

[0061] Step S30: Obtain the video frame to be processed from the second cache.

[0062] Step S40: determining the video frame to be displayed according to the video frame to be processed, and saving the video frame to be displayed into the third buffer.

[0063] In the embodiment of the present invention, the video frames to be processed are taken out from the second buffer, and the video frames to be displayed determined according to the video frames to be processed are saved in the third buffer, so that the video frames in the third buffer are played according to the display time of the video frames.

[0064] It should be noted that whether the video frame to be displayed can be played directly is determined according to the display timestamp. If so, the video frame to be displayed is played. Otherwise, the video frame to be displayed is saved in the third cache and displayed in the third cache in the order of the display timestamp.

[0065] To sum up, the video processing method provided in the embodiment of the present invention dynamically adjusts the size of the first cache, the second cache and the third cache according to the network conditions and business load, and uses the three caches to hierarchically store the received video frames, the video frames to be processed and the video frames to be displayed, effectively avoiding resource contention and resource waste, and optimizing resource utilization, thereby improving the efficiency and reliability of video frame processing and improving the overall stability of the video processing system.

[0066] Optionally, in actual applications, the video frames that meet the video processing conditions can be filtered out in batches based on the cache time, decoding timestamp, and display timestamp of the video frames, effectively reducing the probability of resource preemption. Figure 5 , Figure 4 The sub-steps of step S20 may include:

[0067] Step S201: Move video frames whose decoding timestamps are greater than or equal to the display timestamps from the first cache to the second cache.

[0068] In the embodiment of the present invention, the decoding time stamp (Decoding Time Stamp, referred to as: DTS) is used to indicate the time when the data of the video frame is decoded, and the presentation time stamp (Presentation Time Stamp, referred to as: PTS) is used to indicate the time when the data of the video frame is displayed. The decoding time stamp and the presentation time stamp of each video frame are preset by the network camera.

[0069] Step S202: Move video frames whose actual buffering time exceeds the buffering time threshold from the first buffer to the second buffer.

[0070] In the embodiment of the present invention, after receiving the video frame, the network video recorder records the actual cache time of the video frame and saves the video frame to the first cache. The cache time threshold is pre-set according to the actual application scenario and the processing capacity of the network video recorder, and the cache time threshold supports update adjustment.

[0071] Step S203: When the first cache is empty, the received video frame is saved in the second cache.

[0072] In the implementation of the present invention, when the network video recorder is powered on, a first cache, a second cache and a third cache are created, and the newly created first cache, second cache and third cache are set to an initialization state. At this time, no video frame is saved in the first cache, and the first cache is empty. The received video frame is saved in the second cache to wait for processing. When the first cache saves the video frame, the initialization state is updated to a non-initialization state. When the first cache is in the non-initialization state, the received video frame is saved in the first cache. The initialization state of the second cache and the third cache is updated in a similar manner to the first cache.

[0073] It can be seen that the embodiment of the present invention effectively reduces resource contention by migrating video frames in batches, and can improve the performance of the video processing system in processing video frames and the stability of system operation.

[0074] Optionally, in actual applications, the user may dynamically adjust the order of acquiring video frames in the second cache according to network conditions or service load, thereby ensuring timely transmission and playback of important video frames. Figure 4 The sub-steps of step S30 may include:

[0075] The video frames to be processed are obtained from the second cache in order of priority from high to low; or, the video frames to be processed are obtained from the second cache in sequence.

[0076] As a possible implementation, when the network condition is poor or the business load is high, the network condition or business load is displayed to the user through a visualization interface, and the user is advised to turn on the priority read switch. When the user turns on the priority read switch, the priorities of the video frames in the second cache are traversed, and the video frames to be processed are taken out from the second cache in order of priority. When the network condition is good or the business load is low, the suggestion to turn on the priority read switch in the visualization interface is deleted, and the user can turn off the priority read switch according to the display information of the visualization interface. When the user turns off the priority read switch, the video frames to be processed are taken out from the second cache in sequence according to the order in which the video frames are saved in the second cache.

[0077] In the world of video coding, I-frames, P-frames, and B-frames form the cornerstones of video compression. These frame types directly affect the quality and smoothness of the video. High-priority video frames include I-frames and P-frames. Among them, I-frames, also known as key frames, are complete image frames that can be decoded independently without relying on information from other frames. P-frames are forward predictive coding frames that are generated based on the previous I-frame or P-frame.

[0078] It can be seen that the embodiment of the present invention dynamically adjusts the processing order of video frames according to network conditions or business loads. When the network conditions are poor or the business load is high, high-priority video frames are processed first to reduce resource contention, so that the video processing system can process video frames more effectively, thereby ensuring the timely transmission and dissemination of important video information and improving the efficiency and reliability of video frame processing.

[0079] Optionally, in practical applications, frame skipping may be used to ensure video real-time performance. Figure 4 The sub-step of determining the video frame to be displayed according to the video frame to be processed in step S30 may include:

[0080] When the preset frame skipping switch is turned off, the video frames to be processed are decoded to obtain the video frames to be displayed; when the frame skipping switch is turned on, the video frames to be processed that meet the preset frame skipping interval are decoded to obtain the video frames to be displayed.

[0081] As a possible implementation, when the network condition is poor, the user is advised to turn on the frame skipping switch through the visual interface. When the user turns on the frame skipping switch, the user needs to pre-set the frame skipping interval, which is used to represent the number of frames skipped during the video frame processing. Assuming that the frame skipping interval is 20, every 20 frames of video frames are decoded to obtain the video frame to be displayed, thereby reducing the computational complexity of video frame processing and improving the processing efficiency of video frames.

[0082] When the network condition is good, the suggestion to turn on the frame skipping switch in the visualization interface is deleted, and the user can turn off the frame skipping switch according to the display information of the visualization interface. When the user turns off the frame skipping switch, each video frame to be processed is decoded to obtain a video frame to be displayed.

[0083] Optionally, in practical applications, a frame interpolation method is used to ensure video fluency. The video processing method may include the following steps:

[0084] When the frame supplementation condition is met, a transition frame is generated according to two adjacent video frames in the third cache, and the transition frame is saved in the third cache; the transition frame is used to smooth the motion trajectory between the two adjacent video frames.

[0085] In the embodiment of the present invention, when the network condition is good, the user is advised to turn on the frame interpolation switch through the visual interface. When the user turns on the frame interpolation switch, the frame interpolation condition is met. The video fluency is ensured by generating transition frames.

[0086] As a possible implementation manner, generating a transition frame according to two adjacent video frames in the third buffer includes the following steps:

[0087] First, a motion vector is obtained based on two video frames; the motion vector represents the motion trajectory of each pixel from the previous video frame to the next video frame.

[0088] In the embodiment of the present invention, the two video frames are respectively converted into corresponding grayscale images, and the Farneback optical flow algorithm of OpenCV (e.g., CV2) is used to calculate the motion vectors of the two video frames according to the two grayscale images. Assuming that OpenCV is CV2, the two grayscale images are used as inputs of the cv2.calcOpticalFlowFarneback function to calculate the motion vectors of the two video frames.

[0089] Secondly, the motion vector is interpolated according to the time parameter, and the pixel coordinates of the transition frame are determined according to the interpolated motion vector.

[0090] In the embodiment of the present invention, the time parameter can be set according to actual needs, and the value range of the time parameter is (0, 1). The motion vector is multiplied by the time parameter to obtain the difference motion vector. The grid coordinates are generated according to the position of each pixel in the two video frames, and the grid coordinates are adjusted using the difference motion vector to obtain the pixel coordinates of the two adjacent video frames and the pixel coordinates of the transition frame.

[0091] Finally, the pixel value corresponding to the pixel coordinate is determined according to the pixel values ​​of the corresponding coordinates in the two video frames.

[0092] In an embodiment of the present invention, the pixel values ​​of two adjacent video frames are mapped to the coordinates after the difference using the cv2.remap function of OpenCV, and the pixel values ​​of the two pixel frames are weighted averaged using the cv2.addWeighted function to generate the pixel value of each pixel coordinate in the transition frame.

[0093] Optionally, in actual applications, you can monitor network conditions through bandwidth, latency, jitter, packet loss rate, and throughput, and monitor business load through CPU utilization, memory usage, bandwidth usage, disk read and write speed, and cache usage. Figure 5 , Figure 4 The sub-steps of step S10 may include:

[0094] Step S101, determining a network status score according to bandwidth, delay, jitter, packet loss rate and throughput.

[0095] As a possible implementation, taking a network video recorder supporting 32 video channels as an example, it is assumed that the first cache, the second cache and the third cache are set to 100M, 200M and 100M respectively by default, and the high threshold and the low threshold of the network condition and the business load are pre-set according to the actual application scenario.

[0096] Assume that when the bandwidth or throughput exceeds the high threshold, the bandwidth score or throughput score is set to 2; when the bandwidth or throughput is lower than the high threshold but not lower than the low threshold, the bandwidth score or throughput score is set to 1; when the bandwidth or throughput is lower than the low threshold, the bandwidth score or throughput score is set to 0. Similarly, when the delay, jitter, or packet loss rate does not exceed the low threshold, the delay score, jitter score, or packet loss rate score is set to 2; when the delay, jitter, or packet loss rate is lower than the high threshold but not lower than the low threshold, the delay score, jitter score, or packet loss rate score is set to 2; when the delay, jitter, or packet loss rate exceeds the high threshold, the delay score, jitter score, or packet loss rate score is set to 0.

[0097] You can pre-set high and low thresholds based on actual application scenarios. For example, the high bandwidth threshold is 100 Mbps, the low bandwidth threshold is 10 Mbps, the high delay threshold is 100 milliseconds, the low delay threshold is 50 milliseconds, the high jitter threshold is 20 milliseconds, the low jitter threshold is 5 milliseconds, the high packet loss rate threshold is 5%, the low packet loss rate threshold is 1%, the high throughput threshold is 80% of the bandwidth, and the low throughput threshold is 60% of the bandwidth.

[0098] When bandwidth ≥ 100Mbps, latency ≤ 50ms, jitter ≤ 5ms, packet loss rate ≤ 1%, and throughput ≥ 80%*bandwidth, the bandwidth score, latency score, jitter score, packet loss rate score, and throughput score are each 2, and the accumulated network status score is 10, indicating a good network condition. When making adjustments based on a good network condition, the first cache, second cache, and third cache can be set to 50M, 150M, and 50M, respectively.

[0099] When 10Mbps≤Bandwidth<100Mbps, 50ms≤Delay≤100ms, 5ms<Jitter≤20ms, 1%<Packet Loss Rate≤5%, 60%*Bandwidth≤Throughput<80%*Bandwidth, the bandwidth score, delay score, jitter score, packet loss rate score and throughput score are 1 respectively, and the accumulated network status score is 5, indicating that the network status is general. When making adjustments based on general network status, the first cache, second cache and third cache can be set to 100M, 200M and 100M respectively.

[0100] When bandwidth < 10Mbps, delay > 100ms, jitter > 20ms, packet loss rate > 5%, and throughput < 60% * bandwidth, the bandwidth score, delay score, jitter score, packet loss rate score, and throughput score are all 0, and the accumulated network status score is 0, indicating that the network status is poor. When making adjustments based on the poor network status, the first cache, second cache, and third cache can be set to 200M, 300M, and 200M, respectively.

[0101] Step S102, determining a business load score according to CPU utilization, memory utilization, bandwidth utilization, disk read and write speed, and cache utilization.

[0102] Assume that when the CPU utilization is lower than the low threshold of business load, the CPU utilization score is set to 2, and when the CPU utilization is higher than the high threshold of business load, the CPU utilization score is set to 0. Similarly, the scores of memory usage, bandwidth usage, disk read / write rate, and cache usage are set according to the high and low thresholds.

[0103] For example, the low threshold of business load is preset to 50%, and the high threshold of business load is preset to 80%. When CPU utilization <50%, memory utilization <50%, bandwidth utilization <50%, disk read / write rate <50%, and cache utilization <50%, the CPU utilization score, memory utilization score, bandwidth utilization score, disk read / write rate score, and cache utilization score are 2 respectively, and the accumulated business load score is 10, indicating that the business load is low. When adjusting based on low business load, the first cache, the second cache, and the third cache can be set to 50M, 150M, and 50M respectively.

[0104] When CPU utilization>80%, memory utilization>80%, bandwidth utilization>80%, disk read / write rate>80%, and cache utilization>80%, the CPU utilization score, memory utilization score, bandwidth utilization score, disk read / write rate score, and cache utilization score are all 0, and the accumulated business load score is 0, indicating that the business load is high. When making adjustments based on high business load, the first cache, second cache, and third cache can be set to 100M, 200M, and 200M, respectively.

[0105] Step S103: updating the sizes of the first cache, the second cache, and the third cache according to the network status score and the service load score.

[0106] In the embodiment of the present invention, the first cache, the second cache and the third cache are dynamically adjusted according to the real-time monitored network conditions and service loads, so as to optimize resource utilization and reduce resource contention problems.

[0107] Optionally, in actual applications, the network status and the service load may be scored by setting a threshold policy or a weight policy, and the sizes of the three caches may be dynamically adjusted according to the scores. Figure 5 The sub-steps of step S103 may include:

[0108] Determine a comprehensive score based on the network status score, the network status weight, the business load score and the business load weight, and update the sizes of the first cache, the second cache and the third cache based on the comprehensive score; or, update the sizes of the first cache, the second cache and the third cache based on the worst score of the network status score and the business load score.

[0109] In an embodiment of the present invention, the video processing system supports setting a threshold strategy or a weight strategy. If the user adopts the threshold strategy, the network status score and the business load score are calculated according to the default high and low thresholds or the high and low thresholds set by the user, the network status score and the business load score are compared, and the sizes of the first cache, the second cache, and the third cache are updated according to the worst score.

[0110] Assuming that the bandwidth and throughput are both lower than the low threshold, and the delay, jitter, and packet loss rate are all higher than the high threshold, the network status score is 0, the CPU utilization is higher than the high threshold, and the memory usage, bandwidth usage, disk read / write rate, and cache usage are all lower than the low threshold, then the business load score is 2. Then, according to the worst split (i.e., the network status score), the first cache, the second cache, and the third cache are adjusted to 200M, 300M, and 200M, respectively.

[0111] If the user adopts the weight strategy, the user sets the network status weight and business load weight according to the application scenario. To ensure the real-time quality of the video, the user can increase the network status weight and give priority to the network status. If in a high-load computing scenario, the business load weight can be increased to give priority to the business load.

[0112] The product of the network status weight and the network status score and the product of the service load weight and the service load score are summed to obtain a comprehensive score. The sizes of the first cache, the second cache, and the third cache are adjusted according to the comprehensive score.

[0113] It can be seen that the embodiment of the present invention uses an adaptive algorithm to dynamically adjust the sizes of the first cache, the second cache, and the third cache according to the network condition and the service load, balance the system load, and avoid system overload or resource idleness. When the network condition is good or the service load is low, reducing the cache size can effectively optimize resource utilization. When the network condition is poor or the service load is high, increasing the cache size can reduce the probability of resource contention, thereby improving the overall performance and reliability of the video processing system.

[0114] Based on the same inventive concept, the basic principle and technical effect of the video processing device provided in the embodiment of the present invention are the same as those in the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference may be made to the corresponding contents in the above embodiment.

[0115] Please refer to Figure 6 , Figure 6 The block diagram of the video processing device 200 provided by the embodiment of the present invention is shown. The video processing device 200 is applied to a network video recorder. A first cache, a second cache and a third cache are created on the network video recorder. The first cache is used to store received video frames, the second cache is used to store video frames to be processed, and the third cache is used to store video frames to be displayed. The video processing device 200 includes an adjustment module 201 and a processing module 202.

[0116] An adjustment module 201, configured to adjust the sizes of the first cache, the second cache, and the third cache according to network conditions and service load;

[0117] The processing module 202 is used to move the video frames that meet the video processing conditions from the first cache to the second cache; obtain the video frames to be processed from the second cache; determine the video frames to be displayed according to the video frames to be processed, and save the video frames to be displayed in the third cache.

[0118] To sum up, the video processing device provided in the embodiment of the present invention dynamically adjusts the size of the first cache, the second cache and the third cache according to the network conditions and business load, and uses the three caches to hierarchically store the received video frames, the video frames to be processed and the video frames to be displayed, effectively avoiding resource contention and resource waste, and optimizing resource utilization, thereby improving the efficiency and reliability of video frame processing and improving the overall stability of the video processing system.

[0119] Optionally, the processing module 202 is specifically used to move video frames whose decoding timestamps are greater than or equal to the display timestamps from the first cache to the second cache; move video frames whose actual cache time exceeds the cache time threshold from the first cache to the second cache; and when the first cache is empty, save the received video frames to the second cache.

[0120] Optionally, the processing module 202 is specifically configured to obtain the to-be-processed video frames from the second cache in order of priority from high to low; or, to obtain the to-be-processed video frames from the second cache in sequence.

[0121] Optionally, the processing module 202 is specifically used to decode the video frames to be processed to obtain the video frames to be displayed when the preset frame skipping switch is turned off; when the frame skipping switch is turned on, decode the video frames to be processed that meet the preset frame skipping interval to obtain the video frames to be displayed.

[0122] Optionally, the processing module 202 is specifically configured to generate a transition frame according to two adjacent video frames in the third cache and save the transition frame in the third cache when the frame supplementation condition is met; the transition frame is used to smooth the motion trajectory between two adjacent video frames.

[0123] Optionally, the network condition includes bandwidth, delay, jitter, packet loss rate and throughput; the business load includes CPU utilization, memory utilization, bandwidth utilization, disk read and write speed and cache utilization; the adjustment module 201 is specifically used to determine the network condition score according to the bandwidth, delay, jitter, packet loss rate and throughput; determine the business load score according to the CPU utilization, memory utilization, bandwidth utilization, disk read and write speed and cache utilization; and update the size of the first cache, the second cache and the third cache according to the network condition score and the business load score.

[0124] Optionally, the adjustment module 201 is specifically used to determine a comprehensive score based on the network status score, the network status weight, the business load score and the business load weight, and update the size of the first cache, the second cache and the third cache based on the comprehensive score; or, update the size of the first cache, the second cache and the third cache based on the worst score of the network status score and the business load score.

[0125] Please refer to Figure 7 , is a block diagram of a network video recorder 100 provided in an embodiment of the present invention. The network video recorder 100 includes a memory 110, a processor 120 and a communication module 130. The memory 110, the processor 120 and the communication module 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0126] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0127] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, when the computer program stored in the memory 110 is executed by the processor 120, the video processing method disclosed in the above embodiments can be implemented.

[0128] The communication module 130 is used to establish a communication connection between the network video recorder 100 and other communication terminals through the network, and is used to send and receive data through the network.

[0129] It should be understood that Figure 7 The structure shown is only a schematic diagram of the structure of the network video recorder 100. The network video recorder 100 may also include Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown. Figure 7 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0130] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 120, the video processing method disclosed in the above embodiments is implemented.

[0131] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0133] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video processing method, characterized in that: Applied to a network video recorder, a first cache, a second cache and a third cache are created on the network video recorder; the first cache is used to store received video frames, the second cache is used to store video frames to be processed, and the third cache is used to store video frames to be displayed; the method includes: Adjust the sizes of the first cache, the second cache, and the third cache according to network conditions and business load; The video frames meeting the video processing condition are moved from the first cache to the second cache, including: moving the video frames whose decoding timestamp is greater than or equal to the display timestamp from the first cache to the second cache; moving the video frames whose actual cache time exceeds the cache time threshold from the first cache to the second cache; when the first cache is empty, saving the received video frames to the second cache; Obtaining the to-be-processed video frames from the second cache, including: when the priority reading switch is turned on, traversing the priorities of the video frames in the second cache, and taking out the to-be-processed video frames from the second cache in the order of the priorities; when the priority reading switch is turned off, taking out the to-be-processed video frames from the second cache in sequence according to the order in which the video frames in the second cache are saved; the priority reading switch is turned on by the user when the network condition is poor or the service load is high, and is turned off by the user when the network condition is good or the service load is low; A video frame to be displayed is determined according to the video frame to be processed, and the video frame to be displayed is saved in the third cache.

2. The video processing method according to claim 1, characterized in that: The step of determining the video frame to be displayed according to the video frame to be processed comprises: When the preset frame skipping switch is turned off, decoding the video frame to be processed to obtain the video frame to be displayed; When the frame skipping switch is turned on, the video frames to be processed that meet the preset frame skipping interval are decoded to obtain the video frames to be displayed.

3. The video processing method according to claim 1, characterized in that: The method further comprises: When the frame supplementation condition is met, a transition frame is generated according to two adjacent video frames in the third cache, and the transition frame is saved in the third cache; the transition frame is used to smooth the motion trajectory between the two adjacent video frames.

4. The video processing method according to claim 1, characterized in that: The network conditions include bandwidth, delay, jitter, packet loss rate and throughput; the business load includes CPU utilization, memory utilization, bandwidth utilization, disk read and write speed and cache utilization; The adjusting the sizes of the first cache, the second cache, and the third cache according to the network condition and the service load includes: Determine a network status score according to the bandwidth, the delay, the jitter, the packet loss rate, and the throughput; Determine a business load score according to the CPU utilization, the memory utilization, the bandwidth utilization, the disk read and write speed, and the cache utilization; The sizes of the first cache, the second cache, and the third cache are updated according to the network status score and the service load score.

5. The video processing method according to claim 4, characterized in that: The updating the sizes of the first cache, the second cache, and the third cache according to the network status score and the service load score comprises: Determine a comprehensive score based on the network status score, the network status weight, the business load score and the business load weight, and update the sizes of the first cache, the second cache and the third cache based on the comprehensive score; or, update the sizes of the first cache, the second cache and the third cache based on the worst score of the network status score and the business load score.

6. A video processing device, characterized in that: Applied to a network video recorder, a first cache, a second cache and a third cache are created on the network video recorder; the first cache is used to store received video frames, the second cache is used to store video frames to be processed, and the third cache is used to store video frames to be displayed; the device includes: An adjustment module, configured to adjust the sizes of the first cache, the second cache, and the third cache according to network conditions and service load; A processing module, used for moving video frames whose decoding timestamps are greater than or equal to display timestamps from the first cache to the second cache; moving video frames whose actual cache time exceeds the cache time threshold from the first cache to the second cache; when the first cache is empty, saving the received video frames to the second cache; when the priority read switch is turned on, traversing the priorities of the video frames in the second cache, and taking out the to-be-processed video frames from the second cache in the order of the priorities; when the priority read switch is turned off, taking out the to-be-processed video frames from the second cache in sequence according to the order in which the video frames in the second cache are saved; the priority read switch is turned on by the user when the network condition is poor or the business load is high, and is turned off by the user when the network condition is good or the business load is low; determining the video frames to be displayed based on the video frames to be processed, and saving the video frames to be displayed to the third cache.

7. A network video recorder, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the video processing method described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 5 is implemented.

Images