Method and device for diagnosing abnormal types of multi-frame video pictures and medium

Abstract

The invention discloses a multi-frame video picture abnormity type diagnosis method and device and a medium, and relates to the technical field of monitoring videos. The method comprises the following steps: acquiring monitoring videos to be diagnosed, the number of the monitoring videos and an area where a corresponding camera is located; under the condition that the number of the monitoring videos is greater than a first preset threshold value, dividing the monitoring videos according to areas, and calculating the number of the monitoring videos in each area; under the condition that the number of the monitoring videos in the same area is greater than a second preset threshold value, calling the monitoring videos according to a preset priority; according to diagnosis options preset by a user, a multi-thread algorithm is called to conduct parallel diagnosis on the monitoring video, the abnormity type of the monitoring video is obtained, and the diagnosis options comprise freezing diagnosis, jitter diagnosis and offset diagnosis. According to the method, the monitoring video is diagnosed in parallel by calling the multi-thread algorithm, and the most suitable diagnosis algorithm is selected to analyze the monitoring video according to the diagnosis options preset by the user.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a method, device and medium for diagnosing abnormal types of multi-frame video images. Background Art

[0002] In the field of video surveillance, image quality is of vital importance. High-quality video images can provide clearer and more accurate monitoring information, which helps to promptly detect and deal with potential security issues. The quality of video images in the field of video surveillance is directly related to whether AI or security personnel can promptly detect and warn of risky behaviors or events, and plays a vital role in the real-time nature of monitoring. Moreover, different types of video image quality problems usually have different appearance characteristics, and algorithms need to be designed specifically for the appearance of each abnormal problem.

[0003] However, the diagnosis of video quality is a complex process that requires comprehensive consideration of multiple factors. There is no complete product or method that integrates multiple single-frame no-reference algorithms (single-frame no-reference algorithm is an image processing technology that is mainly used to evaluate the quality or extract features of a single frame image without the original image or other reference images). Different monitoring scenarios and needs have different requirements for video quality. However, integrating these algorithms into one product will not only increase the complexity of the product, but may also affect the performance and stability of the product.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] The video surveillance system in the prior art lacks targeted design for each abnormal problem and different requirements for different scenarios, and does not consider increasing stability after integration. Summary of the invention

[0006] The embodiments of the present application provide a method, a device and a medium, which can solve the problems of the video surveillance system in the prior art, which lacks targeted design for each abnormal problem, different requirements for different scenarios, and does not consider the problem of increasing stability after integration.

[0007] In the first aspect, an embodiment of the present application provides a method for diagnosing abnormal types of multi-frame video images, the method comprising: obtaining surveillance videos to be diagnosed, the number of surveillance videos and the areas where the corresponding cameras are located, the surveillance videos including indoor videos and outdoor videos; when the number of surveillance videos is greater than a first preset threshold, dividing the surveillance videos according to areas, and calculating the number of surveillance videos in each area; when the number of surveillance videos in the same area is greater than a second preset threshold, retrieving the surveillance videos according to a preset priority; according to the user-preset diagnostic options, calling a multi-threaded algorithm to perform parallel diagnosis on the surveillance videos to obtain the abnormal type of the surveillance videos, the diagnostic options including freeze diagnosis, jitter diagnosis and offset diagnosis.

[0008] In one implementation of the present application, based on the user-preset diagnostic options including jitter diagnosis, freeze diagnosis and offset diagnosis, a multi-threaded algorithm is called according to the user-preset diagnostic options to perform parallel diagnosis on the surveillance video, specifically including: selecting two adjacent surveillance image frames in the surveillance video, wherein one surveillance image is a reference frame image and the other surveillance image is a comparison frame image; calling the jitter diagnosis algorithm to calculate the row pixel sum and column pixel sum of the reference frame image and the comparison frame image in thread one respectively; calculating the row pixel mean and the column pixel mean based on the row pixel sum and column pixel sum of the reference frame image and the comparison frame image; calculating the row direction projection and the column direction projection based on the row pixel mean and the column pixel mean; calculating the minimum value of the column direction and row direction mutual correlation information of the reference frame image and the comparison frame image based on the row direction projection and the column direction projection respectively; when the minimum value is greater than the jitter threshold, diagnosing the surveillance video as a jitter abnormality.

[0009] In one implementation of the present application, the method also includes: calling an offset diagnosis algorithm to perform a color gamut transformation from a BGR image to a grayscale image on a reference frame image and a comparison frame image in thread two; performing key point detection on the reference frame image and the comparison frame image respectively through a SURF detector to obtain key points; matching the key points based on a FlannBasedMatcher matcher to obtain a first feature point; filtering the first feature point using a RANSAC algorithm to remove mismatched key points to obtain a second feature point; calculating the offset sum of the 2D physical coordinates of the second feature point, wherein the offset sum is the offset sum of the reference frame image and the comparison frame image; calculating an offset mean based on the offset sum to obtain an offset; and when the offset exceeds an offset threshold, diagnosing the surveillance video as an offset abnormality.

[0010] In one implementation of the present application, the method also includes: calling the freezing diagnosis algorithm in thread three to perform a color gamut transformation of the reference frame image and the comparison frame image from BGR image to HSV image; based on the HSV image, respectively calculating the histogram features of the reference frame image and the comparison frame image, and normalizing the histogram features; calculating the similarity of the histogram features of the reference frame image and the comparison frame image to obtain a similarity score; if the similarity score is higher than a third preset threshold, the counter increments the preset score to obtain a count value; when the count value is greater than the freezing threshold, the monitoring video is diagnosed as a freezing abnormality.

[0011] In one implementation of the present application, after calling a multi-threaded algorithm to perform parallel diagnosis on the surveillance video according to the diagnostic options preset by the user and obtaining the abnormality type of the surveillance video, the method also includes: when it is diagnosed as a jitter abnormality, introducing meteorological data and air pressure data; when it is diagnosed as an offset abnormality, notifying the staff to conduct on-site investigation; when it is diagnosed as a freezing abnormality, marking the camera and performing a secondary diagnosis.

[0012] In one implementation of the present application, when a jitter abnormality is diagnosed, meteorological data and air pressure data are introduced, specifically including: for outdoor cameras, based on historical meteorological data and records of jitter abnormalities, learning the changing pattern of jitter abnormalities, the meteorological data includes wind direction and wind speed; dynamically adjusting the jitter threshold according to the changing pattern.

[0013] In one implementation of the present application, after calling a multi-threaded algorithm to perform parallel diagnosis on the surveillance video according to the diagnostic options preset by the user and obtaining the abnormal type of the surveillance video, the method also includes: storing the abnormal type in a cache and returning it to the upper-level user for visual display; providing the user with the option of customizing the diagnostic options of the surveillance video and changing the preset threshold.

[0014] In one implementation of the present application, when the number of surveillance videos in the same area is greater than a second preset threshold, a first preset number of surveillance videos in the same surveillance area is retrieved according to a preset priority, specifically including: sorting the area and the cameras in the same area according to the preset priority; when the number of surveillance videos in the area or the same area is greater than a second preset threshold, retrieving the first preset number of surveillance videos in the area or the same surveillance area according to the preset priority.

[0015] In a second aspect, an embodiment of the present application also provides a diagnostic device for abnormal types of multi-frame video images, the device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain surveillance videos to be diagnosed, as well as the number of surveillance videos and the areas where the corresponding cameras are located, the surveillance videos including indoor videos and outdoor videos; when the number of surveillance videos is greater than a first preset threshold, divide the surveillance videos according to areas, and calculate the number of surveillance videos in each area; when the number of surveillance videos in the same area is greater than a second preset threshold, retrieve the surveillance videos according to a preset priority; according to the diagnostic options preset by the user, call a multi-threaded algorithm to perform parallel diagnosis on the surveillance videos to obtain the abnormal type of the surveillance videos, the diagnostic options including freeze diagnosis, jitter diagnosis and offset diagnosis.

[0016] In the third aspect, the embodiment of the present application also provides a non-volatile computer storage medium for diagnosing the abnormal type of a multi-frame video screen, storing computer executable instructions, and the computer executable instructions are set to: obtain the surveillance video to be diagnosed, as well as the number of surveillance videos and the area where the corresponding cameras are located, the surveillance videos including indoor videos and outdoor videos; when the number of surveillance videos is greater than a first preset threshold, divide the surveillance videos according to the area, and calculate the number of surveillance videos in each area; when the number of surveillance videos in the same area is greater than a second preset threshold, retrieve the surveillance videos according to the preset priority; according to the user-preset diagnostic options, call a multi-threaded algorithm to perform parallel diagnosis on the surveillance videos to obtain the abnormal type of the surveillance videos, and the diagnostic options include freeze diagnosis, jitter diagnosis and offset diagnosis.

[0017] The embodiments of the present application provide a method, device and medium for diagnosing the abnormal types of multiple-frame video images. The method calls a multi-threaded algorithm to perform parallel diagnosis on the surveillance video, reasonably utilizes computing resources, prevents the same video from repeatedly occupying internal test resources, significantly shortens the diagnosis time, improves processing efficiency, and allows users to obtain analysis results as soon as possible; according to the user's preset diagnosis options, the most appropriate diagnosis algorithm is selected to analyze the surveillance video; in the diagnosis of jitter abnormalities, by introducing meteorological data and air pressure data, the jitter threshold can also be dynamically adjusted, allowing the user to change the preset threshold, and divide the surveillance video and camera into areas to adapt to different monitoring environments and requirements, reducing false alarms and missed alarms caused by environmental factors; by selectively processing surveillance videos according to priority, the system can utilize these resources more effectively and avoid resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 A flowchart of a method for diagnosing abnormal types of multiple video frames provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of the internal structure of a diagnostic device for abnormal types of multiple-frame video images provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] The embodiments of the present application provide a method, device and medium for diagnosing abnormal types of multi-frame video images, which solves the problem that the video surveillance system in the prior art lacks targeted design for each abnormal problem, different requirements for different scenarios, and does not consider increasing stability after integration.

[0023] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A flow chart of a method for diagnosing abnormal types of multiple video frames provided in an embodiment of the present application. Figure 1 As shown, a method for diagnosing abnormal types of multiple video frames provided in an embodiment of the present application specifically includes the following steps:

[0025] Step 10: Obtain the surveillance video to be diagnosed, as well as the number of surveillance videos and the area where the corresponding cameras are located. The surveillance videos include indoor videos and outdoor videos.

[0026] In this step, the surveillance video is divided into indoor video and outdoor video for example. In actual operation, more targeted distinctions can be made, such as shopping malls, construction sites, and traffic intersections. Different thresholds can be configured by the user for different scenes.

[0027] Step 20: When the number of surveillance videos is greater than a first preset threshold, the surveillance videos are divided according to regions, and the number of surveillance videos in each region is calculated;

[0028] In this step, if the number of videos to be diagnosed is too large, they can be divided according to the monitoring area, for example, into the parking lot in area B on the first floor and the parking lot in area A on the second floor.

[0029] Step 30: When the number of surveillance videos in the same area is greater than a second preset threshold, the surveillance videos are retrieved according to a preset priority;

[0030] As an optional embodiment, when the number of surveillance videos in the same area is greater than the second preset threshold, retrieving a first preset number of surveillance videos in the same surveillance area according to a preset priority may specifically include: Step 301: sorting the areas and cameras in the same area according to the preset priority;

[0031] In this step, for example, the parking lot in Area B on the first floor is divided into 30 parking spaces, and there are 15 cameras for these 30 parking spaces. Assuming that the parking lot in Area B on the first floor and the entrance have the highest risk level, the parking lot in each area and the 15 cameras in each area are numbered and sorted according to the risk level. The actual operation will be based on the actual situation.

[0032] Step 302: When the number of surveillance videos in a region or the same region is greater than a second preset threshold, a first preset number of surveillance videos in the region or the same surveillance region are retrieved according to a preset priority.

[0033] In this step, if the second preset threshold of the area is 10, and the second preset threshold of the same area is 5, and there are 12 areas, and some of them have more than 5 surveillance videos to be diagnosed, the top 10 risk levels and the top 5 cameras in the area are first retrieved for diagnosis. In other words, it is determined whether the abnormality is caused by a single camera or a camera in a specific area.

[0034] Step 40: According to the diagnosis options preset by the user, a multi-thread algorithm is called to perform parallel diagnosis on the surveillance video to obtain the abnormal type of the surveillance video. The diagnosis options include freeze diagnosis, jitter diagnosis and offset diagnosis.

[0035] In this step, users can customize diagnostic options. For example, due to strong winds outdoors today, the camera will inevitably shake. Therefore, the jitter diagnosis option can be configured for the outdoor camera today. After obtaining a surveillance video to be diagnosed, freeze diagnosis and offset diagnosis are performed on the video at the same time. This can make rational use of computing resources, perform parallel analysis on the same video frame, prevent the same video from repeatedly occupying internal test resources, and achieve accelerated computing, so that users can obtain analysis results as soon as possible.

[0036] It is understandable that the data collected in the production environment is usually a YUV data stream, which may be different from the encoding format required by the actual algorithm. Therefore, it is necessary to use the Opencv tool to perform color gamut space transformation to map the original data stream to the BGR color space.

[0037] As an optional embodiment, based on the user-preset diagnosis options including jitter diagnosis, freeze diagnosis and offset diagnosis, calling a multi-thread algorithm to perform parallel diagnosis on the surveillance video according to the user-preset diagnosis options may specifically include: step 401: selecting two adjacent surveillance image frames in the surveillance video, wherein one surveillance image frame is a reference frame image, and the other surveillance image frame is a comparison frame image;

[0038] In this step, video image jitter refers to the unstable shaking of the captured image caused by the camera being affected by wind, vibration or other external interference or the camera itself being installed unstably, which will cause certain distortion or drift of the video image. Therefore, it is necessary to determine whether there is an image jitter problem by comparing the features between consecutive video frames. The image jitter detection algorithm inputs a multi-frame algorithm, which considers the comparison between adjacent video frames.

[0039] Step 402: calling the jitter diagnosis algorithm, and calculating the row pixel sum and column pixel sum of the reference frame image and the comparison frame image in thread 1 respectively;

[0040] In this step, the main process parses the data and the types of exceptions that need to be screened, and multiple threads call different algorithms to achieve parallel scheduling and screening of multiple types of exceptions, meeting the real-time requirements when multiple types of exceptions that need to be screened are issued. Targeted algorithm design is carried out based on the statistical characteristics of different types of exception algorithms, and feature comparison between multiple frames of images is used to determine whether there are related exceptions; two adjacent frames of images are selected, one as a reference frame srcImg, and the other as a comparison frame baseImg, and each row and column of the image is traversed and pixel values ​​are accumulated.

[0041] Step 403: Calculate the row pixel mean and column pixel mean according to the row pixel sum and column pixel sum of the reference frame image and the comparison frame image;

[0042] In this step, the row pixel mean and the column pixel mean are obtained by dividing the row and column pixel sums by the number of pixels in the row and column.

[0043] Step 404: Calculate the row direction projection and the column direction projection according to the row pixel mean and the column pixel mean;

[0044] In this step, the pixel mean of each row or column is compared with the pixel mean of the adjacent row or column, and the difference between them is calculated. The pixel sum of each row in the row pixel sum arrays Rsum_src and Rsum_base is replaced by the difference between the pixel sum of the row and the mean of the row pixel sum. The column-wise projections of the two frames of images are calculated respectively, and the pixel sum of each column in the column pixel sum arrays Csum_src and Csum_base is replaced by the difference between the pixel sum of the column and the mean of the column pixel sum.

[0045] Step 405: Calculate the minimum value of the column-direction and row-direction mutual correlation information of the reference frame image and the comparison frame image based on the row-direction projection and the column-direction projection respectively;

[0046] Step 406: When the minimum value is greater than the jitter threshold, diagnose the surveillance video as having jitter abnormality.

[0047] In this step, the minimum values ​​of the column-wise and row-wise cross-correlation information of the two frames of images are calculated based on the row-wise projection and the column-wise projection, respectively. If the minimum column-wise cross-correlation value or the minimum row-wise cross-correlation value is greater than a threshold, it is considered that a jitter anomaly exists. For example, for a sequence of 10 frames, if such anomalies occur 5 or more times, it is considered that this sequence has a screen jitter anomaly.

[0048] As an optional embodiment, the method may further include: Step 401': calling the offset diagnosis algorithm to perform color gamut transformation from BGR image to grayscale image on the reference frame image and the comparison frame image in thread 2;

[0049] In this step, video image offset refers to a large deviation of the camera position resulting in a large change in the entire video image, and the captured image is not the scene image to be monitored; image offset requires the advance configuration of the reference frame under the current camera, and the recognition process is mainly achieved through the matching difference between the feature points of the camera's real-time image and the reference frame.

[0050] Step 402': perform key point detection on the reference frame image and the comparison frame image respectively through a SURF detector to obtain key points;

[0051] In this step, the SURF (Speeded Up Robust Features) detector is used to detect key points in the image. These key points are usually significant and stable features in the image, such as corners, edges, or texture-rich areas.

[0052] Step 403': Match the key points based on the FlannBasedMatcher matcher to obtain the first feature point;

[0053] In this step, we have images and their respective key points, but we still need to know how these key points correspond, that is, use FlannBasedMatcher (fast nearest neighbor search algorithm matcher) to match the key points in the reference frame and the comparison frame.

[0054] Step 404': Filter the first feature points using the RANSAC algorithm to remove mismatched key points to obtain second feature points;

[0055] In this step, the matches may not all be correct due to changes in lighting conditions, so the RANSAC (Random Sample Consensus) algorithm is used to remove incorrect matches.

[0056] Step 405 ′: Calculate the offset sum of the 2D physical coordinates of the second feature point, wherein the offset sum is the offset sum of the reference frame image and the comparison frame image;

[0057] In this step, for each pair of matched and retained key points, the difference in their positions on the image is calculated.

[0058] Step 406': Calculate the offset mean according to the offset sum to obtain the offset;

[0059] In this step, in order to obtain a more stable offset estimate, the total offset is divided by the number of matching points to obtain an offset mean, which is the basis for whether an offset occurs.

[0060] Step 407 ′: when the offset exceeds the offset threshold, diagnose the monitoring video as having an offset abnormality.

[0061] In this step, the calculated offset mean and a preset offset threshold are calculated. If the offset mean is greater than the threshold, an offset anomaly exists.

[0062] As an optional embodiment, the method may further include: Step 401″: calling the freezing diagnosis algorithm in thread three to perform color gamut transformation of the reference frame image and the comparison frame image from BGR image to HSV image;

[0063] In this step, the HSV color space consists of three components: hue, saturation, and value. It is more suitable for human visual perception and image processing tasks than the BGR color space.

[0064] Step 402 ”: Based on the HSV image, the histogram features of the reference frame image and the comparison frame image are calculated respectively, and the histogram features are normalized;

[0065] In this step, the histogram is a statistical representation of the image, which shows the distribution of different pixel values ​​(or color components) in the image. In order to make the histogram features more stable and comparable under different lighting conditions, these histogram features are normalized and the feature values ​​are scaled to a specific range (usually 0 to 1) to eliminate the differences between different images due to factors such as lighting and contrast.

[0066] Step 403": Calculate the similarity of the histogram features of the reference frame image and the comparison frame image to obtain a similarity score;

[0067] In this step, the similarity between the reference frame image and the comparison frame image can be evaluated by using histogram intersection, chi-square distance, correlation coefficient, etc. The similarity score is a value between 0 and 1, which indicates the similarity between the two images in color distribution. The higher the score, the more similar the two images are; the lower the score, the greater the difference between the two images.

[0068] Step 404": if the similarity score is higher than the third preset threshold, the counter is incremented by the preset score to obtain the count value; Step 405": when the count value is greater than the freezing threshold, the monitoring video is diagnosed as being frozen abnormally.

[0069] In this step, video freezing refers to the situation where the video screen freezes, which may be caused by a camera data transmission failure; the freezing detection algorithm is a multi-frame algorithm, which considers the comparison between adjacent video frames. For example, the histogram similarity of two frames is calculated to obtain a similarity score. If the similarity score is higher than the freezing threshold, the counter is incremented by 1. If the counter count value is greater than 5, it is considered that there is an abnormal video freezing problem.

[0070] As an optional embodiment, after calling a multi-thread algorithm to perform parallel diagnosis on the monitoring video according to the diagnosis options preset by the user and obtaining the abnormal type of the monitoring video, the method may further include: Step 50: In the case of being diagnosed as a jitter abnormality, introducing meteorological data and air pressure data;

[0071] In this step, analyze whether there are vibration sources in the camera environment, such as nearby machinery running, vehicles passing by, etc., and check whether the camera is affected by wind, especially in outdoor environments. If the image jitter is caused by a camera hardware failure, such as a damaged lens or sensor failure, it needs to be repaired or replaced accordingly.

[0072] As an optional embodiment, when a jitter abnormality is diagnosed, meteorological data and air pressure data are introduced, which may specifically include: Step 501: For outdoor cameras, based on historical meteorological data and records of jitter abnormalities, learn the changing pattern of jitter abnormalities, where the meteorological data includes wind direction and wind speed; Step 502: Dynamically adjust the jitter threshold according to the changing pattern.

[0073] In this step, for example, in strong winds, the jitter threshold can be increased to reduce false alarms caused by weather conditions. This dynamic adjustment can improve the accuracy and robustness of jitter anomaly detection.

[0074] Step 50': in case of being diagnosed as abnormal deviation, notify the staff to conduct on-site investigation;

[0075] In this step, the offset abnormality may mean that the position of the camera has changed. For example, there are branches, construction equipment, etc. around that have collided with the camera. Through on-site investigation, the staff can check the installation position and fixing method of the camera to determine the specific cause of the offset abnormality and take corresponding repair measures.

[0076] Step 50": In the case of being diagnosed as freezing abnormality, the camera is marked and a secondary diagnosis is performed.

[0077] In this step, the camera is marked for subsequent tracking and processing; the secondary diagnosis is a more in-depth analysis and confirmation of the freezing anomaly, which may include checking the camera's power supply, network connection, communication lines, software settings, etc. to determine the specific cause of the freezing anomaly, such as a problem with the communication between the camera and the monitoring center; for another example, for a camera that is initially diagnosed as abnormal at a certain time, the camera is regularly checked, and the user can actively choose whether to start the regular secondary diagnosis function.

[0078] It is understandable that the secondary diagnosis is not only for freezing abnormalities, and the staff's on-site inspection is not only for offset abnormalities. In the embodiment of the present application, only some problems and countermeasures are listed, and the staff can arrange and combine them by themselves during actual operation.

[0079] As an optional embodiment, after calling a multi-threaded algorithm to perform parallel diagnosis on the surveillance video according to the user's preset diagnostic options and obtaining the abnormal type of the surveillance video, the method may also include: Step 60: storing the abnormal type in the cache and returning it to the upper-level user for visual display; Step 70: providing the user with the option of customizing the diagnostic options of the surveillance video and changing the preset threshold.

[0080] In this step, for example, the user can choose to detect only jitter anomalies without detecting offset or freeze anomalies; or the user can choose to use a more advanced diagnostic algorithm to improve the accuracy of the diagnosis; the user can also change the jitter threshold, offset threshold or freeze threshold to adapt to different monitoring environments and requirements.

[0081] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides a multi-frame video screen abnormality type diagnosis device, whose structure is as follows Figure 2 shown.

[0082] Figure 2 A schematic diagram of the internal structure of a multi-frame video image abnormality diagnosis device provided in an embodiment of the present application. Figure 2 As shown, the device includes:

[0083] at least one processor 201;

[0084] and, a memory 202 communicatively connected to the at least one processor;

[0085] Among them, the memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201 so that at least one processor 201 can: obtain surveillance videos to be diagnosed, as well as the number of surveillance videos and the areas where the corresponding cameras are located, the surveillance videos include indoor videos and outdoor videos; when the number of surveillance videos is greater than a first preset threshold, divide the surveillance videos according to areas, and calculate the number of surveillance videos in each area; when the number of surveillance videos in the same area is greater than a second preset threshold, retrieve the surveillance videos according to a preset priority; according to the user-preset diagnostic options, call a multi-threaded algorithm to perform parallel diagnosis on the surveillance videos to obtain the abnormal type of the surveillance videos, and the diagnostic options include freeze diagnosis, jitter diagnosis and offset diagnosis.

[0086] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for diagnosing abnormal types of multi-frame video images stores computer executable instructions, and the computer executable instructions are set to: obtain surveillance videos to be diagnosed, as well as the number of surveillance videos and the area where the corresponding cameras are located, the surveillance videos include indoor videos and outdoor videos; when the number of surveillance videos is greater than a first preset threshold, divide the surveillance videos according to areas, and calculate the number of surveillance videos in each area; when the number of surveillance videos in the same area is greater than a second preset threshold, retrieve the surveillance videos according to the preset priority; according to the user's preset diagnosis options, call a multi-threaded algorithm to perform parallel diagnosis on the surveillance videos to obtain the abnormal type of the surveillance videos, the diagnosis options include freeze diagnosis, jitter diagnosis and offset diagnosis.

[0087] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0088] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0089] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0094] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0095] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0097] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for diagnosing abnormal types of multi-frame video images, characterized in that: The method comprises: Obtaining surveillance videos to be diagnosed, as well as the number of surveillance videos and the areas where corresponding cameras are located, wherein the surveillance videos include indoor videos and outdoor videos; When the number of the surveillance videos is greater than a first preset threshold, the surveillance videos are divided according to the regions, and the number of surveillance videos in each region is calculated; When the number of surveillance videos in the same area is greater than a second preset threshold, the surveillance videos are retrieved according to a preset priority; According to the diagnosis options preset by the user, a multi-thread algorithm is called to perform parallel diagnosis on the monitoring video to obtain the abnormal type of the monitoring video, and the diagnosis options include freeze diagnosis, jitter diagnosis and offset diagnosis.

2. The method for diagnosing abnormal types of multi-frame video images according to claim 1, characterized in that: In the case where the user-preset diagnostic options include jitter diagnosis, freeze diagnosis, and offset diagnosis, calling a multi-threaded algorithm to perform parallel diagnosis on the surveillance video according to the user-preset diagnostic options specifically includes: Selecting two adjacent surveillance image frames in the surveillance video, wherein one surveillance image frame is a reference frame image and one surveillance image frame is a comparison frame image; Call the jitter diagnosis algorithm, and calculate the row pixel sum and column pixel sum of the reference frame image and the comparison frame image in thread 1 respectively; The row pixel mean and the column pixel mean are calculated according to the row pixel sum and column pixel sum of the reference frame image and the comparison frame image; A row direction projection and a column direction projection are calculated according to the row pixel mean and the column pixel mean; Calculating minimum values ​​of column-direction and row-direction mutual correlation information of the reference frame image and the comparison frame image based on the row-direction projection and the column-direction projection respectively; When the minimum value is greater than the jitter threshold, the monitoring video is diagnosed as having abnormal jitter.

3. The method for diagnosing abnormal types of multi-frame video images according to claim 2, characterized in that: The method further comprises: Calling the offset diagnosis algorithm, in thread 2, performing a color gamut transformation from a BGR image to a grayscale image on the reference frame image and the comparison frame image; Performing key point detection on the reference frame image and the comparison frame image respectively by using a SURF detector to obtain key points; Matching the key points based on the FlannBasedMatcher matcher to obtain the first feature point; Using the RANSAC algorithm to filter the first feature points, remove the mismatched key points, and obtain the second feature points; Calculating the offset sum of the 2D physical coordinates of the second feature point, wherein the offset sum is the offset sum of the reference frame image and the comparison frame image; Calculate the offset mean according to the offset sum to obtain the offset; When the offset exceeds an offset threshold, the monitoring video is diagnosed as having an offset abnormality.

4. The method for diagnosing abnormal types of multi-frame video images according to claim 2, characterized in that: The method further comprises: The freezing diagnosis algorithm is called in thread three to perform color gamut conversion from BGR image to HSV image on the reference frame image and the comparison frame image; Based on the HSV image, the histogram features of the reference frame image and the comparison frame image are respectively calculated, and the histogram features are normalized; Calculating the similarity of the histogram features of the reference frame image and the comparison frame image to obtain a similarity score; If the similarity score is higher than a third preset threshold, the counter is incremented by a preset score to obtain a count value; When the count value is greater than the freezing threshold, the monitoring video is diagnosed as being frozen abnormally.

5. The method for diagnosing abnormal types of multi-frame video images according to claim 1, characterized in that: After invoking a multi-thread algorithm to perform parallel diagnosis on the surveillance video according to the diagnosis options preset by the user and obtaining the abnormal type of the surveillance video, the method further includes: In the case of abnormal jitter, meteorological data and air pressure data are introduced; In the event of a diagnosed deviation anomaly, notify staff to conduct an on-site investigation; In the event of a freezing anomaly being diagnosed, the camera is marked and a secondary diagnosis is performed.

6. The method for diagnosing abnormal types of multi-frame video images according to claim 1, characterized in that: In the case of abnormal jitter, meteorological data and air pressure data are introduced, including: For outdoor cameras, learn the changing pattern of abnormal jitter based on historical meteorological data and records of abnormal jitter, where the meteorological data includes wind direction and wind speed; The jitter threshold is dynamically adjusted according to the change rule.

7. The method for diagnosing abnormal types of multi-frame video images according to claim 6, characterized in that: After invoking a multi-thread algorithm to perform parallel diagnosis on the surveillance video according to the diagnosis options preset by the user and obtaining the abnormal type of the surveillance video, the method further includes: The exception type is stored in a cache and returned to the upper-level user for visual display; Provide users with diagnostic options to customize surveillance videos, option to change preset thresholds.

8. The method for diagnosing abnormal types of multi-frame video images according to claim 1, characterized in that: When the number of surveillance videos in the same area is greater than the second preset threshold, a first preset number of surveillance videos in the same surveillance area are retrieved according to the preset priority, specifically including: Sorting the area and cameras in the same area according to a preset priority; When the number of surveillance videos in the area or the same area is greater than the second preset threshold, a first preset number of surveillance videos in the area or the same surveillance area are retrieved according to the preset priority.

9. A diagnostic device for abnormal types of multi-frame video images, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Obtaining surveillance videos to be diagnosed, as well as the number of surveillance videos and the areas where corresponding cameras are located, wherein the surveillance videos include indoor videos and outdoor videos; When the number of the surveillance videos is greater than a first preset threshold, the surveillance videos are divided according to the regions, and the number of surveillance videos in each region is calculated; When the number of surveillance videos in the same area is greater than a second preset threshold, a first preset number of surveillance videos in the same surveillance area are retrieved according to a preset priority; According to the diagnosis options preset by the user, a multi-thread algorithm is called to perform parallel diagnosis on the monitoring video to obtain the abnormal type of the monitoring video, and the diagnosis options include freeze diagnosis, jitter diagnosis and offset diagnosis.

10. A non-volatile computer storage medium for diagnosing abnormal types of multi-frame video images, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Obtaining surveillance videos to be diagnosed, as well as the number of surveillance videos and the areas where corresponding cameras are located, wherein the surveillance videos include indoor videos and outdoor videos; When the number of the surveillance videos is greater than a first preset threshold, the surveillance videos are divided according to the regions, and the number of surveillance videos in each region is calculated; When the number of surveillance videos in the same area is greater than a second preset threshold, a first preset number of surveillance videos in the same surveillance area are retrieved according to a preset priority; According to the diagnosis options preset by the user, a multi-thread algorithm is called to perform parallel diagnosis on the monitoring video to obtain the abnormal type of the monitoring video, and the diagnosis options include freeze diagnosis, jitter diagnosis and offset diagnosis.

Images