Real-time video analysis method and device and storage medium

Abstract

The invention discloses a real-time video analysis method and device and a storage medium, and the real-time video analysis method comprises the steps: caching video frames collected in real time to obtain a video frame cache sequence, and carrying out the motion detection of the video frame cache sequence to obtain a motion detection result; acquiring a video analysis sampling frame rate set for each time period in advance, and determining a sampling frame rate corresponding to the time period of the current time to obtain a basic sampling frame rate; adjusting the basic sampling frame rate based on the motion detection result to obtain a target sampling frame rate; and extracting video frames from the video frame buffer sequence according to the target sampling frame rate, and analyzing the extracted video frames to obtain an analysis result. By combining the current time period and the motion detection result, the sampling frame rate of video frame analysis is comprehensively set, calculation resources are remarkably saved, meanwhile, no-missing analysis of high-value video frames is ensured, and the video analysis effect is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a real-time video analysis method, device and storage medium. Background Technology

[0002] Traditional video analytics relies heavily on manual labor, which is inefficient and has low accuracy. With the development of big data technology, existing video acquisition systems have also evolved to include real-time video content analysis capabilities. By analyzing real-time captured video content, functions such as event recording, event response, and target detection can be performed.

[0003] While continuous analysis of the video at full frame rate and across the entire screen achieves comprehensive video analysis, it can overload the computing power at the camera's edge. Therefore, a fixed frame interval is generally used to perform frame-by-frame analysis on the video content. However, this may lead to the omission of key information and result in low analysis efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides at least one real-time video analysis method, device, and storage medium.

[0005] The first aspect of this application provides a real-time video analysis method, which includes: buffering real-time acquired video frames to obtain a video frame buffer sequence; performing motion detection on the video frame buffer sequence to obtain motion detection results; obtaining a sampling frame rate for video analysis pre-set for each time period; determining the sampling frame rate corresponding to the current time period to obtain a basic sampling frame rate; adjusting the basic sampling frame rate based on the motion detection results to obtain a target sampling frame rate; extracting video frames from the video frame buffer sequence according to the target sampling frame rate; analyzing the extracted video frames to obtain analysis results.

[0006] In one embodiment, adjusting the base sampling frame rate based on motion detection results to obtain a target sampling frame rate includes: obtaining the motion level of the current scene based on the motion detection results; if the motion level is greater than a preset motion level, increasing the base sampling frame rate to obtain the target sampling frame rate; if the motion level is not greater than the preset motion level, directly using the base sampling frame rate as the target sampling frame rate.

[0007] In one embodiment, adjusting the base sampling frame rate based on motion detection results to obtain a target sampling frame rate includes: obtaining a motion frame rate mapping table; obtaining a sampling frame rate matching the motion detection results from the motion frame rate mapping table to obtain a motion sampling frame rate; wherein the motion frame rate mapping table stores sampling frame rates corresponding to different motion detection results; and obtaining the target sampling frame rate based on the base sampling frame rate and the motion sampling frame rate.

[0008] In one embodiment, obtaining the target sampling frame rate based on the base sampling frame rate and the motion sampling frame rate includes: acquiring the accuracy corresponding to the motion detection result; determining the weight parameters of the base sampling frame rate and the motion sampling frame rate based on the accuracy; and performing a weighted summation of the base sampling frame rate and the motion sampling frame rate based on the weight parameters to obtain the target sampling frame rate.

[0009] In one embodiment, adjusting the base sampling frame rate based on motion detection results to obtain a target sampling frame rate includes: adjusting the base sampling frame rate based on motion detection results to obtain an initial adjusted frame rate; obtaining the computing resource load; if the computing resource load exceeds a preset load threshold, reducing the initial adjusted frame rate to obtain the target sampling frame rate; if the computing resource load does not exceed the preset load threshold, directly using the initial adjusted frame rate as the target sampling frame rate.

[0010] In one embodiment, the extracted video frames are analyzed to obtain analysis results, including: detecting whether there is an event in the extracted video frames; if there is an event, then extracting video frames from the video frame buffer sequence using a preset high sampling frame rate, and continuing to detect events in the extracted video frames to obtain analysis results.

[0011] In one embodiment, the method further includes: obtaining a matching data storage strategy based on the analysis results; and storing the extracted video frames and / or video frame cache sequences based on the data storage strategy.

[0012] In one embodiment, based on the analysis results, a matching data storage strategy is obtained, including: if the analysis results indicate that no event exists, the matching data storage strategy is: recording the metadata of the extracted video frames and / or low-frequency video snapshots; if the analysis results indicate that an event exists, the matching data storage strategy is: extracting the original video frames associated with the event from the video frame cache sequence to obtain the original event video, and storing the original event video.

[0013] The second aspect of this application provides a real-time video analysis device, which includes: a motion detection module, used to buffer real-time acquired video frames to obtain a video frame buffer sequence, and to perform motion detection on the video frame buffer sequence to obtain a motion detection result; The base frame rate determination module is used to obtain the sampling frame rate of video analysis that is pre-set for each time period, and determine the sampling frame rate corresponding to the time period at that time to obtain the base sampling frame rate. The target frame rate determination module is used to adjust the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate; The video frame extraction and analysis module is used to extract video frames from the video frame buffer sequence according to the target sampling frame rate, analyze the extracted video frames, and obtain the analysis results.

[0014] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described real-time video analysis method.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described real-time video analysis method.

[0016] The above scheme obtains a video frame cache sequence by caching real-time acquired video frames, performs motion detection on the video frame cache sequence to obtain motion detection results, acquires the sampling frame rate for video analysis pre-set for each time period, determines the sampling frame rate corresponding to the current time period to obtain the base sampling frame rate, adjusts the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate, extracts video frames from the video frame cache sequence according to the target sampling frame rate, analyzes the extracted video frames to obtain analysis results, and comprehensively sets the sampling frame rate for video frame analysis by combining the current time period and motion detection results. This significantly saves computing resources while ensuring the analysis of all high-value video frames without omission, thus improving the video analysis effect.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a flowchart illustrating a real-time video analysis method in an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating the sampling frequencies corresponding to multiple time periods in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of a sampling configuration file shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating video data storage as shown in an exemplary embodiment of this application; Figure 5 This is a block diagram illustrating a real-time video analysis apparatus according to an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0022] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] The real-time video analysis method provided in the embodiments of this application will be described below.

[0024] The real-time video analysis method provided in this application can be executed by a video acquisition device, such as a camera that supports Always-on (AO) technology, like an Always-on Record (AOR) camera. Alternatively, the execution entity for each step can be other electronic devices communicatively connected to the video acquisition device, or the real-time video analysis method can be executed through interaction between the video acquisition device and other electronic devices. This application does not limit the scope of the method.

[0025] Please see Figure 1 , Figure 1 This is a flowchart illustrating a real-time video analysis method in an exemplary embodiment of this application, as shown below. Figure 1 As shown, the real-time video analysis method includes at least steps S110 to S140, which are described in detail below: Step S110: Buffer the real-time acquired video frames to obtain a video frame buffer sequence, and perform motion detection on the video frame buffer sequence to obtain motion detection results.

[0026] Acquire real-time video frames and write them into a circular buffer to obtain a video frame buffer sequence.

[0027] Motion detection is performed on the video frame buffer sequence to obtain motion detection results. The motion detection results are used to reflect the differences between adjacent video frames. They can be the degree of specific scene changes, the scene type (such as static scene, dynamic scene), or the presence of moving targets.

[0028] For example, methods such as inter-frame interpolation, optical flow, and deep learning can be used to perform motion detection on the video frame buffer sequence to obtain motion detection results.

[0029] For example, the pixel difference between adjacent video frames is calculated. If the pixel difference is larger, the motion detection result indicates a greater degree of scene change; if the pixel difference is smaller, the motion detection result indicates a smaller degree of scene change.

[0030] Optionally, to avoid increasing the computational resource load, a lightweight algorithm can be used to perform motion detection on the video frame buffer sequence, such as using the inter-frame interpolation method, which requires less computational resources, to achieve motion detection.

[0031] Step S120: Obtain the sampling frame rate of the video analysis that is pre-set for each time period, and determine the sampling frame rate corresponding to the time period at that time to obtain the basic sampling frame rate.

[0032] Multiple time periods can be divided using units such as a day, a week, a month, or a year as the basic unit.

[0033] The sampling frame rate for video analysis is pre-set for each time period.

[0034] Users can set the corresponding sampling frame rate for each time period based on experience, or they can flexibly calculate the corresponding sampling frame rate for each time period based on the actual situation. This application does not limit this.

[0035] Then, determine the time period at that time, obtain the sampling frame rate corresponding to that time period, and get the base sampling frame rate.

[0036] Step S130: Adjust the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate.

[0037] Based on the current motion detection results and the current adapted base sampling frame rate, the target sampling frame rate is determined.

[0038] For example, the degree of scene change is obtained based on the motion detection results, a specific frame rate adjustment value is determined based on the degree of scene change, and the base sampling frame rate is adjusted based on the frame rate adjustment value to obtain the target sampling frame rate.

[0039] For example, the adjustment strategy for the base sampling frame rate is determined based on the motion detection results. This strategy can include adjusting upwards, downwards, or not adjusting. If the motion detection results indicate that the current scene is static, the adjustment strategy is no adjustment; if the motion detection results indicate that the current scene is transitioning from static to dynamic, the adjustment strategy is upwards; and if the motion detection results indicate that the current scene is transitioning from dynamic to static, the adjustment strategy is downwards. The specific frame rate adjustment value can be determined based on the degree of scene change or can be a preset value.

[0040] Of course, in addition to the sampling frame rate adjustment method in the example above, other adjustment methods can be used, and this application does not limit them.

[0041] Step S140: Extract video frames from the video frame buffer sequence according to the target sampling frame rate, analyze the extracted video frames, and obtain the analysis results.

[0042] Video frames are extracted from the video frame buffer sequence according to the target sampling frame rate. The higher the target sampling frame rate, the more video frames are extracted, and the lower the target sampling frame rate, the fewer video frames are extracted.

[0043] The extracted video frames are analyzed to obtain the analysis results.

[0044] The analysis methods can include: detecting the existence of an event, identifying the type of the event, detecting the existence of a target object, identifying the type of the target object, and extracting the trajectory of the target object. The specific analysis method can be set according to the actual needs of the scenario.

[0045] The real-time video analysis method provided in this application obtains a video frame cache sequence by caching real-time acquired video frames, and performs motion detection on the video frame cache sequence to obtain motion detection results; it acquires the sampling frame rate for video analysis pre-set for each time period, determines the sampling frame rate corresponding to the current time period to obtain the base sampling frame rate; it adjusts the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate; it extracts video frames from the video frame cache sequence according to the target sampling frame rate, analyzes the extracted video frames to obtain analysis results, and comprehensively sets the sampling frame rate for video frame analysis by combining the current time period and motion detection results. This significantly saves computing resources while ensuring the analysis of all high-value video frames without omission, thus improving the video analysis effect.

[0046] The following describes some embodiments of this application in detail.

[0047] The sampling frame rate for video analysis is pre-set for each time period to form a sampling configuration file, which is then loaded during video analysis.

[0048] The sampling profile can be automatically generated by learning from historical data.

[0049] For example, the analysis results of video frames corresponding to each time period within a historical time period are obtained to obtain historical analysis results. These results include event recognition results and / or target recognition results. Based on these historical analysis results, the number of event recognition results and / or target recognition results for each time period within the historical time period are counted to obtain the number of successfully detected events and / or targets for each time period within the historical time period. Based on these numbers, the sampling frame rate for video analysis in each time period is set. Specifically, the higher the number of successfully detected events and / or targets within a time period, the higher the sampling frame rate for that time period; conversely, the lower the number of successfully detected events and / or targets within a time period, the lower the sampling frame rate for that time period.

[0050] By statistically analyzing historical results, the sampling frame rate of video analysis for each time period is accurately set, and a sampling configuration file is generated to configure differentiated parameters for different time periods, thereby improving the rationality of computing resource allocation.

[0051] In addition, in practical applications, the sampling frame rate of video analysis for each time period can be updated synchronously based on the updates of historical analysis results, so that the calculated target sampling frame rate can be continuously adapted to the actual application scenario.

[0052] For examples, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the sampling frequencies corresponding to multiple time periods in an exemplary embodiment of this application, as shown below. Figure 2 As shown, using a 24-hour day as the basic unit of division, four time periods are obtained, including the late night period, the morning peak period, the daytime off-peak period, and the transition period. Each time period has a corresponding sampling frequency.

[0053] For example, statistical analysis of historical data shows that the number of successfully detected events and targets is low during late-night hours, so a lower sampling frequency is set, sampling once every 60 seconds; during morning rush hours, the number of successfully detected events and targets is high, so a higher sampling frequency is set, sampling once every 5 seconds, and so on.

[0054] In one implementation, to improve the accuracy of the sampling frame rate configuration, the sampling configuration file can set multiple basic sampling frame rates for a single time period, with different sampling frame rates adapting to different scenarios. For example, for the same time period, different sampling frame rates can be set for different weather conditions and / or for different light intensities.

[0055] The sampling frame rate can be set by the user based on experience for different weather conditions and / or different light intensities, or it can be obtained by statistically analyzing historical data recorded under different weather conditions and / or different light intensities. This application does not limit this.

[0056] During real-time video analysis, the base sampling frame rate is obtained by querying the matching sampling configuration file based on the current time period, and / or the current weather, and / or the current light intensity.

[0057] For an example illustrating the sampling configuration file using a specific application scenario, please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of a sampling configuration file shown in an exemplary embodiment of this application, as follows: Figure 3 As shown, based on the current time period, current weather, and current light intensity, the sampling configuration file is retrieved. This sampling configuration file consists of multiple fields, such as unique identifier, effective time period, basic sampling frame rate, scene applicable conditions, priority, etc.

[0058] Priority indicates the order of multiple sampling profiles. This priority can be set by the user based on experience, or it can be based on the accuracy of each sampling profile, with higher accuracy resulting in higher priority. The accuracy of a sampling profile can be obtained by detecting events and / or targets after applying that profile.

[0059] If multiple sampling configuration files are matched at the same time, the sampling configuration file with the highest priority can be selected.

[0060] Of course, the sampling configuration file can also consist of more or fewer fields, depending on the actual application scenario.

[0061] An example is given to illustrate how to adjust the sampling frame rate.

[0062] In Example 1, step S130 adjusts the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate, including the following steps S1311 to S1314.

[0063] Step S1311: Obtain the motion level of the current scene based on the motion detection results.

[0064] Step S1312: Determine whether the exercise level is greater than the preset exercise level. If yes, proceed to step S1313; otherwise, proceed to step S1314.

[0065] Step S1313: Increase the base sampling frame rate to obtain the target sampling frame rate.

[0066] Step S1314: Directly use the base sampling frame rate as the target sampling frame rate.

[0067] The motion detection result specifically represents the motion level of the current scene. The greater the difference between adjacent video frames, the higher the motion level; the smaller the difference, the lower the motion level.

[0068] When the motion level is greater than the preset motion level, it indicates that there is significant motion in the current scene. At this time, the base sampling frame rate is immediately increased to analyze more video frames and avoid missing information in the analysis.

[0069] The specific value to be added can be determined based on the specific exercise level; the higher the exercise level, the higher the value to be added. The specific value to be added can also be a pre-set empirical value, which is not limited in this application.

[0070] When the motion level is no greater than the preset motion level, it means that there is no significant motion in the current scene. In this case, the basic sampling frame rate is maintained to save computing resources.

[0071] In Example 2, step S130 adjusts the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate, including the following steps S1321 to S1322.

[0072] Step S1321: Obtain the motion frame rate mapping table, and obtain the sampling frame rate that matches the motion detection result from the motion frame rate mapping table to obtain the motion sampling frame rate; wherein, the motion frame rate mapping table stores the sampling frame rate corresponding to different motion detection results.

[0073] Different sampling frame rates can be set for different motion detection results to obtain a motion frame rate mapping table. For example, when the motion detection result indicates a static scene, the corresponding sampling frame rate is 1 frame every 60 seconds; when the motion detection result indicates a dynamic scene, the corresponding sampling frame rate is 1 frame every 5 seconds.

[0074] Specifically, different sampling frame rates can be set for different motion detection results at different time periods. This can be done by the user setting the sampling frame rate corresponding to different motion detection results at different time periods based on experience; or by statistically analyzing historical records associated with different motion detection results at different time periods to obtain the sampling frame rate corresponding to different motion detection results at different time periods. This application does not limit this approach.

[0075] For example, during late-night hours, if motion detection results show no change in adjacent video frames, the corresponding sampling frame rate is 1 frame every 60 seconds; if motion detection results show changes in adjacent video frames greater than a threshold, the corresponding sampling frame rate is 1 frame every 5 seconds. During morning rush hour, if motion detection results show no change in adjacent video frames, the corresponding sampling frame rate is 1 frame every 20 seconds; if motion detection results show changes in adjacent video frames greater than a threshold, the corresponding sampling frame rate is 1 frame every 2 seconds.

[0076] Step S1322: Obtain the target sampling frame rate based on the base sampling frame rate and the motion sampling frame rate.

[0077] The target sampling frame rate is determined by combining the base sampling frame rate and the motion sampling frame rate.

[0078] You can choose either the base sampling frame rate or the motion sampling frame rate, or you can perform a weighted summation of the base sampling frame rate and the motion sampling frame rate to obtain the target sampling frame rate.

[0079] For example, obtain the remaining available computing resources, determine the matching frame rate calculation strategy based on the size of the remaining available computing resources, and calculate the basic sampling frame rate and motion sampling frame rate based on the frame rate calculation strategy to obtain the target sampling frame rate.

[0080] For example, if the remaining available computing resources are greater than the maximum resource threshold, the maximum value between the base sampling frame rate and the motion sampling frame rate is selected as the target sampling frame rate; if the remaining available computing resources are less than the maximum resource threshold but greater than the minimum resource threshold, the base sampling frame rate and the motion sampling frame rate are weighted and summed to obtain the target sampling frame rate; if the remaining available computing resources are less than the minimum resource threshold, the minimum value between the base sampling frame rate and the motion sampling frame rate is selected as the target sampling frame rate.

[0081] The above embodiments can extract more video frames for in-depth analysis when there are abundant computing resources, thus achieving effective utilization of computing resources. When computing resources are scarce and the system load is high, fewer video frames can be extracted for in-depth analysis, thus avoiding increasing the system load.

[0082] Based on the above embodiments, when performing a weighted summation of the base sampling frame rate and the motion sampling frame rate, the weight parameters used can be preset by the user based on experience, or they can be flexibly calculated according to the actual scenario.

[0083] For example, obtain the accuracy corresponding to the motion detection results; based on the accuracy, determine the weight parameters of the base sampling frame rate and the motion sampling frame rate; based on the weight parameters, perform a weighted summation of the base sampling frame rate and the motion sampling frame rate to obtain the target sampling frame rate.

[0084] Specifically, if the accuracy of the current motion detection result is higher, the weight of the motion sampling frame rate is higher and the weight of the base sampling frame rate is lower; if the accuracy of the current motion detection result is lower, the weight of the motion sampling frame rate is lower and the weight of the base sampling frame rate is higher.

[0085] By determining the weight parameters used for weighted summation of the base sampling frame rate and the motion sampling frame rate based on the accuracy of the motion detection results, we can avoid incorrect sampling frame rate adjustments caused by misidentification and improve the overall stability of the solution.

[0086] In Example 3, step S130 adjusts the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate, including the following steps S1331 to S1332.

[0087] Step S1331: Adjust the base sampling frame rate based on the motion detection results to obtain the initial adjusted frame rate.

[0088] The adjustment method can be found in Embodiments 1 and 2 above, and will not be repeated here.

[0089] Step S1332: Obtain the computing resource load. If the computing resource load exceeds the preset load threshold, reduce the initial adjustment frame rate to obtain the target sampling frame rate. If the computing resource load does not exceed the preset load threshold, use the initial adjustment frame rate directly as the target sampling frame rate.

[0090] Computational resource load is used to measure the amount of work or pressure that a system is currently performing. It can be reflected by metrics such as CPU utilization, memory usage, disk I / O rate, and network bandwidth utilization.

[0091] If the computational resource load exceeds the preset load threshold, it indicates that there are few remaining available computational resources. In this case, the initial adjustment frame rate should be reduced to avoid excessive system load. If the computational resource load does not exceed the preset load threshold, it indicates that there are many remaining available computational resources. In this case, the initial adjustment frame rate should be directly used as the target sampling frame rate to ensure the video frame analysis density.

[0092] After extracting video frames from the video frame buffer sequence according to the target sampling frame rate, the analysis can be performed on the entire extracted video frame or on a portion of the extracted video frame.

[0093] For example, obtain the remaining available computing resources and / or motion detection results, and based on the remaining available computing resources and / or motion detection results, determine whether to analyze all the frames of the extracted video or to analyze a portion of the extracted video frames.

[0094] For example, if the remaining available computing resources are greater than the preset resource threshold, and / or the motion detection results show that the differences between adjacent video frames are large, then all frames of the video frame are extracted for analysis; if the remaining available computing resources are not greater than the preset resource threshold, and / or the motion detection results show that the differences between adjacent video frames are small, then a portion of the video frame is extracted for analysis. Specifically, the region of interest in the video frame can be extracted. The region of interest can be a pre-defined fixed region, or it can be a region in the video frame with large pixel changes.

[0095] The above embodiments can use computing resources to analyze video frame content with higher information value, thereby improving the utilization rate of computing resources and the effect of video analysis.

[0096] In one embodiment, step S140 involves analyzing the extracted video frames to obtain analysis results, including the following steps S141 to S143.

[0097] Step S141: Detect whether there is an event in the extracted video frame. If there is, proceed to step S142; if not, proceed to step S143.

[0098] Step S142: Extract video frames from the video frame buffer sequence using a preset high sampling frame rate, and continue to perform event detection on the extracted video frames to obtain the analysis results.

[0099] If an event is detected after in-depth analysis of video frames, video frames are immediately extracted and analyzed at a high sampling frame rate to ensure that key information can be analyzed and responded to in a timely manner.

[0100] Step S143: Then maintain the original target sampling frame rate or convert it to the base sampling frame rate.

[0101] If no event is detected after in-depth analysis of the video frames, then sampling and analysis will be performed at the original target sampling frame rate or converted to the basic sampling frame rate.

[0102] In one embodiment, the method further includes: obtaining a matching data storage strategy based on the analysis results; and storing the extracted video frames and / or video frame cache sequences based on the data storage strategy.

[0103] Based on the actual analysis results, corresponding data storage methods are adopted for the extracted video frames and / or video frame cache sequences.

[0104] For example, please see Figure 4 , Figure 4 This is a flowchart illustrating video data storage as shown in an exemplary embodiment of this application, such as... Figure 4 As shown, it specifically includes: Step S401: Perform event analysis on the extracted video frames to obtain the analysis results; Step S402: Does an event exist? If not, proceed to step S403; if yes, proceed to steps S404 to S405.

[0105] Step S403: Record the metadata of the extracted video frames and / or low-frequency video snapshots.

[0106] Step S404: Extract the original video frames associated with the event from the video frame buffer sequence to obtain the original event video, and store the original event video.

[0107] Specifically, if the analysis results indicate the existence of an event, the video frame in which the event is first detected is marked, the original video frames N seconds before that video frame are obtained, and the original video frames in which the event exists and the original video frames M seconds after that are continuously obtained to obtain the original event video.

[0108] Step S405: Generate and provide event notification information and event summary.

[0109] For example, sending event notifications and event summaries to the user's device.

[0110] If the analysis results indicate that no event exists, the matching data storage strategy is to record the metadata of the extracted video frames and / or low-frequency video snapshots, storing only the key data to save storage space. If the analysis results indicate that an event exists, the matching data storage strategy is to extract the original video frames associated with the event from the video frame cache sequence to obtain the original event video, and store the original event video to retain the key video data.

[0111] To illustrate the method of this application, let's take a specific application scenario as an example: The plug-in AOR camera includes a video acquisition module, a dynamic intelligent scheduler (including a strategy management unit, a scene perception unit, and a decision execution unit), a load feedback unit, a video analysis engine, and a data management unit.

[0112] The video capture module is configured to capture uninterrupted full-frame-rate high-definition video streams.

[0113] The dynamic intelligent scheduler is communicatively coupled with the video acquisition module, and includes: a scene perception unit: running a lightweight algorithm to perform real-time and fast motion detection on the input video stream and outputting motion detection results; a policy management unit: maintaining and dynamically loading sampling configuration files; and a decision execution unit: combining the output of the scene perception unit and the currently active sampling configuration file to determine the target sampling frame rate, extracting video frames according to the target sampling frame rate, and outputting the video frame sequence to be analyzed.

[0114] Load feedback unit: Real-time detection of computing resource load, and adaptive adjustment of the target sampling frame rate of the current strategy when the computing resource load exceeds the threshold, in order to maintain system stability.

[0115] Video analytics engine: It is communicatively coupled with the dynamic intelligent scheduler and is configured to perform high-precision video analysis tasks on the video frames sent by the dynamic intelligent scheduler.

[0116] Data Management Module: Communicates with the video analytics engine and is configured to execute differentiated storage strategies based on event detection results.

[0117] Specifically, when the AOR camera is powered on, the video acquisition module begins to acquire data at full frame rate and writes it to the loop buffer, thus obtaining a video frame buffer sequence.

[0118] The current time is 2:00 AM. The policy management unit loads the configuration file corresponding to the late-night period based on the current time, and obtains a basic sampling frame rate of 1 frame every 60 seconds.

[0119] The scene perception unit performs lightweight motion detection on video frames in the video frame buffer sequence. The motion detection results show that the motion level is less than the preset motion level.

[0120] The decision execution unit maintains the basic sampling frame rate to extract video frames and sends them to the video analysis engine for in-depth analysis. At this time, the camera load is extremely low.

[0121] When a frame is analyzed and identified as an "illegally parked vehicle" event, the decision execution unit immediately instructs a switch to a high sampling frame rate, and then extracts video frames from the subsequent video stream at a high sampling frame rate to achieve continuous analysis of the event.

[0122] After an event is detected, the data management unit marks the metadata corresponding to the event (such as event type, occurrence time, occurrence location, etc.), extracts the original video frames from N seconds (e.g., 10 seconds) before the event to M seconds (e.g., 5 seconds) after the event from the video frame cache sequence of the circular cache, and combines them into the original event video for permanent storage.

[0123] The sampling frame rate of video analysis can be updated and optimized for each time period based on historical analysis results.

[0124] For example, by periodically (e.g., weekly) analyzing historical analysis results, if it is found that "7:00-9:00 am on weekdays" is the commuting peak and events occur frequently, a high sampling frame rate (e.g., the basic sampling frame rate is 1 frame every 2 seconds) is automatically created for this period to achieve self-optimization of the sampling strategy.

[0125] Figure 5 This is a block diagram illustrating a real-time video analysis apparatus according to an exemplary embodiment of this application. Figure 5 As shown, the exemplary real-time video analytics device 500 includes: The motion detection module 510 is used to buffer real-time acquired video frames to obtain a video frame buffer sequence, and to perform motion detection on the video frame buffer sequence to obtain motion detection results. The base frame rate determination module 520 is used to obtain the sampling frame rate of video analysis that is pre-set for each time period, and determine the sampling frame rate corresponding to the time period at that time to obtain the base sampling frame rate. The target frame rate determination module 530 is used to adjust the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate; The video frame extraction and analysis module 540 is used to extract video frames from the video frame buffer sequence according to the target sampling frame rate, analyze the extracted video frames, and obtain the analysis results.

[0126] It should be noted that the real-time video analysis device and the real-time video analysis method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the real-time video analysis device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation.

[0127] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating the structure of an electronic device in an exemplary embodiment of this application. The electronic device 600 includes a memory 601 and a processor 602. The processor 602 executes program instructions stored in the memory 601 to implement the steps in any of the above-described real-time video analysis method embodiments. In a specific implementation scenario, the electronic device 600 may be a video acquisition device, or it may be other electronic devices communicatively connected to the video acquisition device, such as a microcomputer, server, laptop computer, tablet computer, etc., without limitation herein.

[0128] Specifically, processor 602 controls itself and memory 601 to implement the steps in any of the above-described real-time video analysis method embodiments. Processor 602 can also be referred to as a Central Processing Unit (CPU). Processor 602 may be an integrated circuit chip with signal processing capabilities. Processor 602 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 602 can be implemented using integrated circuit chips.

[0129] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of a computer-readable storage medium in an exemplary embodiment of this application. The computer-readable storage medium 700 stores program instructions 710 that can be executed by a processor. The program instructions 710 are used to implement the steps in any of the above-described real-time video analysis method embodiments.

[0130] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0131] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of real-time video analytics, characterized by, The method includes: The real-time acquired video frames are cached to obtain a video frame cache sequence, and motion detection is performed on the video frame cache sequence to obtain a motion detection result; Obtain the sampling frame rate of the video analysis that is pre-set for each time period, and determine the sampling frame rate corresponding to the time period at that time to obtain the basic sampling frame rate; The base sampling frame rate is adjusted based on the motion detection results to obtain the target sampling frame rate; Video frames are extracted from the video frame buffer sequence according to the target sampling frame rate, and the extracted video frames are analyzed to obtain the analysis results.

2. The method of claim 1, wherein, The step of adjusting the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate includes: The motion level of the current scene is obtained based on the motion detection results; If the motion level is greater than the preset motion level, the base sampling frame rate is increased to obtain the target sampling frame rate; If the motion level is not greater than the preset motion level, then the base sampling frame rate is directly used as the target sampling frame rate.

3. The method of claim 1, wherein, The step of adjusting the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate includes: Obtain a motion frame rate mapping table, and retrieve the sampling frame rate that matches the motion detection result from the motion frame rate mapping table to obtain the motion sampling frame rate; wherein, the motion frame rate mapping table stores the sampling frame rate corresponding to different motion detection results; The target sampling frame rate is obtained based on the base sampling frame rate and the motion sampling frame rate.

4. The method of claim 3, wherein, The process of obtaining the target sampling frame rate based on the base sampling frame rate and the motion sampling frame rate includes: Obtain the accuracy corresponding to the motion detection results; Based on the accuracy, determine the weight parameters for the base sampling frame rate and the motion sampling frame rate; Based on the weight parameters, the base sampling frame rate and the motion sampling frame rate are weighted and summed to obtain the target sampling frame rate.

5. The method of claim 1, wherein, The step of adjusting the base sampling frame rate based on the motion detection results to obtain the target sampling frame rate includes: The base sampling frame rate is adjusted based on the motion detection results to obtain an initial adjusted frame rate; The computing resource load is obtained. If the computing resource load exceeds a preset load threshold, the initial adjusted frame rate is reduced to obtain the target sampling frame rate. If the computing resource load does not exceed the preset load threshold, the initial adjusted frame rate is directly used as the target sampling frame rate.

6. The method of claim 1, wherein, The analysis of the extracted video frames yields the following results: Detect whether an event exists in the extracted video frame; If an event exists, video frames are extracted from the video frame buffer sequence using a preset high sampling frame rate, and event detection is performed on the extracted video frames to obtain the analysis results.

7. The method of claim 1, wherein, The method further includes: Based on the analysis results, a matching data storage strategy is obtained; Based on the data storage strategy, the extracted video frames and / or the video frame cache sequence are stored.

8. The method of claim 7, wherein, The step of obtaining a matching data storage strategy based on the analysis results includes: If the analysis results indicate that no event exists, the matching data storage strategy is to record the metadata and / or low-frequency video snapshots of the extracted video frames. If the analysis results indicate the existence of an event, the matching data storage strategy is as follows: extract the original video frame associated with the event from the video frame cache sequence to obtain the original event video, and store the original event video.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images