A security video monitoring fault detection method and device, a terminal device, and a storage medium

Abstract

This invention discloses a method, apparatus, terminal device, and storage medium for fault detection in security video surveillance. When the similarity between a first video frame acquired at a previous detection time and a second video frame acquired at the current detection time is less than a first threshold, the method checks whether the monitoring device receives a position adjustment command during the interval between the previous and current detection times. If not, it determines that the monitoring device has an image acquisition fault. If so, it determines the theoretical monitoring position that the monitoring device should be set at each time point during the interval based on the position adjustment command; it generates a theoretical image path based on each theoretical monitoring position; and it generates an actual image path based on the actual video frames acquired by the monitoring device. If the theoretical image path and the actual image path are consistent, it determines that the monitoring device has not malfunctioned; otherwise, it determines that the monitoring device has a movement fault. Implementing this invention can improve the accuracy of fault detection in monitoring equipment.

Description

Technical Field

[0001] This invention relates to the field of security monitoring technology, and in particular to a method, device, terminal equipment and storage medium for detecting security video surveillance faults. Background Technology

[0002] Surveillance equipment enables remote monitoring and plays a significant role in ensuring safety in people's daily lives. Fault detection in surveillance equipment is generally based on the images captured by the equipment.

[0003] In existing technologies, the similarity between video frames from two different acquisition points is generally compared to determine whether there is a sudden change in the image between the two video frames. If a sudden change in the image occurs, it indicates that a malfunction has occurred.

[0004] However, the above fault diagnosis method is too simplistic. In real-world scenarios, a monitoring device often needs to monitor multiple areas. Therefore, it is sometimes necessary to periodically move the monitoring device to monitor other areas. The movement of the monitoring device inevitably leads to sudden changes in the monitoring screen. Using the above fault diagnosis method, it is easy to conclude that the monitoring device has malfunctioned after this normal scheduling operation, resulting in a misjudgment. Summary of the Invention

[0005] This invention provides a method, device, terminal equipment, and storage medium for fault detection in security video surveillance, which can improve the accuracy of fault diagnosis in monitoring equipment.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a method for detecting faults in security video surveillance, comprising:

[0007] The system acquires a first video frame captured by the monitoring device at the previous detection time and a second video frame captured at the current detection time; if the first similarity between the first video frame and the second video frame is less than a first threshold, the system detects whether the monitoring device receives a position adjustment command during the interval between the previous detection time and the current detection time.

[0008] If not, then the monitoring equipment is confirmed to have an image acquisition failure;

[0009] If so, the theoretical monitoring positions that the monitoring equipment should be set at each time point in the interval period are determined according to the position adjustment command; the video frames at the corresponding positions are identified from the preset stitched images according to each theoretical monitoring position to obtain each first standard video frame; the center point of each first standard video frame is extracted, and the center points of each first standard video frame are connected in chronological order according to the time corresponding to the theoretical monitoring position to generate the theoretical image path that the monitoring equipment should correspond to when it moves according to the position adjustment command; wherein, the video frames captured by the monitoring equipment at all monitoring positions in the historical period are stitched together according to the corresponding monitoring positions to generate a stitched image.

[0010] Calculate the similarity between each actual video frame collected by the monitoring equipment and each video frame in the stitched image at each time point during the interval period. Take the video frame with the highest similarity in the stitched image as the second standard video frame corresponding to the actual video frame. Extract the center point of each second standard video frame and connect the center points of each second standard video frame in chronological order to generate the actual image path corresponding to the actual movement of the monitoring equipment.

[0011] Determine whether the theoretical image path and the actual image path are consistent. If they are consistent, it is determined that the monitoring equipment is not faulty; if they are inconsistent, it is determined that the monitoring equipment has a movement fault.

[0012] Furthermore, the similarity between the first video frame and the second video frame is calculated in the following manner:

[0013] Determine the static image region and dynamic image region of the first video frame;

[0014] A three-dimensional coordinate system is constructed with the row coordinates of the pixel in the image as the X-axis, the column coordinates of the pixel in the image as the Y-axis, and the pixel value of the pixel as the Y-axis.

[0015] The pixels in the static image area of ​​the first video frame are projected into a three-dimensional coordinate system to generate the first spatial projection.

[0016] The pixels in the dynamic image region of the first video frame are projected into a three-dimensional coordinate system to generate a second spatial projection.

[0017] Based on the static image area of ​​the first video frame, the corresponding pixels in the second video frame are projected into the three-dimensional coordinate system to generate a third spatial projection.

[0018] Based on the dynamic image region of the first video frame, the corresponding pixels in the second video frame are projected into the three-dimensional coordinate system to generate the fourth spatial projection.

[0019] Calculate the similarity between the first spatial projection and the third spatial projection to obtain the static similarity; calculate the similarity between the second spatial projection and the fourth spatial projection to obtain the dynamic similarity.

[0020] A comprehensive similarity is calculated based on the static similarity, the first preset weight corresponding to the static similarity, the dynamic similarity, and the second preset weight corresponding to the dynamic similarity; wherein, the first preset weight is greater than the second preset weight.

[0021] The overall similarity is used as the similarity between the first video frame and the second video frame.

[0022] Furthermore, determining the static image region and dynamic image region of the first video frame includes:

[0023] Based on the first standard image corresponding to the first video frame in the stitched image, the monitoring position of the monitoring device at the previous detection time is determined, and the target monitoring position is obtained.

[0024] Acquire historical video frames collected by the monitoring equipment at the target monitoring location at each historical moment; generate historical video groups based on each historical video frame; wherein each historical video group contains two historical video frames that are temporally adjacent.

[0025] For each historical video group, each historical video frame within the historical video group is divided into several tiles. The second similarity between the tiles corresponding to two historical video frames is calculated one by one. The region corresponding to the tile group with the second similarity less than the second threshold is taken as the dynamic region to be selected for the historical video group.

[0026] The intersection of the selected dynamic regions corresponding to all historical video groups is used to obtain the dynamic image region of the first video frame.

[0027] All image regions in the first video frame, excluding dynamic image regions, are considered as static image regions.

[0028] Furthermore, if it is determined that the monitoring device has a movement fault, the path point where the difference between the theoretical image path and the actual image path begins to appear is determined based on the overlap between the theoretical image path and the actual image path, and the difference path point is obtained.

[0029] Based on the time and location corresponding to the video frame where the difference path point is located, determine the time and location when the monitoring equipment experiences a movement failure.

[0030] Furthermore, after determining that the monitoring device has a mobility fault, the method also includes:

[0031] Determine the positions of the first standard video frames corresponding to the previous detection time and the current detection time in the stitched image, and obtain the first position and the second position respectively;

[0032] Based on the first position and the second position, in the stitched image, when determining the image path from the first position to the second position that avoids the difference path points and has the shortest path, the target image path is obtained;

[0033] The monitoring equipment is controlled to move sequentially according to the monitoring positions corresponding to the target image path to complete the position correction.

[0034] Furthermore, in cases where it is determined that the monitoring equipment has experienced an image acquisition failure, the method also includes:

[0035] Extract the RGB values ​​of each pixel in the second video frame;

[0036] If the RGB values ​​of all pixels are all zero, then the monitoring device is confirmed to have a black screen fault.

[0037] If the RGB value of all pixels is 255, then the monitoring device is confirmed to have a white screen fault.

[0038] Otherwise, it could indicate that the monitoring equipment has been illegally moved or subjected to an image tampering attack.

[0039] Furthermore, after confirming that the monitoring equipment is experiencing a black screen malfunction, the following steps are also taken:

[0040] Based on each actual video frame actually collected by the monitoring equipment at each time interval, determine whether there is a person or object continuously approaching the monitoring equipment.

[0041] If so, the black screen failure is determined to be caused by malicious human obstruction; otherwise, the black screen failure is determined to be caused by damage to the monitoring equipment.

[0042] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments;

[0043] An embodiment of the present invention provides a security video surveillance fault detection device, comprising: an instruction determination module, a first fault identification module, a theoretical image path generation module, an actual image path generation module, and a second fault identification module;

[0044] The instruction determination module is used to acquire the first video frame collected by the monitoring device at the previous detection time and the second video frame collected at the current detection time; if it is determined that the first similarity between the first video frame and the second video frame is less than the first threshold, it detects whether the monitoring device receives a position adjustment instruction during the interval between the previous detection time and the current detection time.

[0045] The first fault identification module is used to determine that the monitoring equipment has an image acquisition fault when it is determined that the monitoring equipment has not received a position adjustment instruction;

[0046] The theoretical image path generation module is used to determine the theoretical monitoring positions that the monitoring equipment should set at each time point in the interval period when the monitoring equipment receives a position adjustment instruction; to identify the video frames at the corresponding positions from the preset stitched images according to each theoretical monitoring position, to obtain each first standard video frame, to extract the center point of each first standard video frame, and to connect the center points of each first standard video frame in chronological order according to the time corresponding to the theoretical monitoring position, thereby generating the theoretical image path that the monitoring equipment should correspond to when it moves according to the position adjustment instruction; wherein, the video frames captured by the monitoring equipment at all monitoring positions in the historical period are stitched together according to the corresponding monitoring positions to generate a stitched image.

[0047] The actual image path generation module is used to calculate the similarity between each actual video frame actually collected by the monitoring device and each video frame in the stitched image at each time of the interval period. The video frame with the highest similarity in the stitched image is used as the second standard video frame corresponding to the actual video frame. The center point of each second standard video frame is extracted, and the center points of each second standard video frame are connected in chronological order to generate the actual image path corresponding to the actual movement of the monitoring device.

[0048] The second fault identification module is used to determine whether the theoretical image path and the actual image path are consistent. If they are consistent, it is determined that the monitoring equipment has not malfunctioned. If they are inconsistent, it is determined that the monitoring equipment has a movement fault.

[0049] Based on the above method embodiments, the present invention provides corresponding terminal device embodiments;

[0050] One embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the security video surveillance fault detection method according to any one of the present invention.

[0051] Based on the above method embodiments, the present invention provides corresponding storage medium embodiments;

[0052] One embodiment of the present invention provides a storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to execute the security video surveillance fault detection method according to any one of the present invention.

[0053] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0054] This invention provides a method, apparatus, terminal device, and storage medium for detecting faults in security video surveillance. The method, when determining that the similarity between a first video frame acquired at a previous detection time and a second video frame acquired at the current detection time is less than a first threshold, checks whether the monitoring device receives a position adjustment command during the interval between the previous and current detection times. If not, it determines that the monitoring device has an image acquisition fault; if so, it determines the theoretical monitoring position that the monitoring device should set at each time point during the interval based on the position adjustment command; it identifies the corresponding video frame from a preset stitched image based on each theoretical monitoring position to obtain each first standard video frame; it extracts the center point of each first standard video frame and connects the center points of each first standard video frame in chronological order according to the time corresponding to the theoretical monitoring position to generate the theoretical image path corresponding to the movement of the monitoring device according to the position adjustment command; wherein, the video frames captured by the monitoring device at all monitoring positions during the historical time period are... The process involves stitching the images together according to their corresponding monitoring locations to generate a stitched image. The similarity between each actual video frame captured by the monitoring equipment and each video frame in the stitched image is calculated at each time interval. The video frame with the highest similarity in the stitched image is taken as the second standard video frame corresponding to the actual video frame. The center points of each second standard video frame are extracted, and the center points are connected sequentially to generate the actual image path corresponding to the actual movement of the monitoring equipment. The process then determines whether the theoretical image path and the actual image path are consistent. If they are consistent, the monitoring equipment is considered to be functioning correctly; otherwise, a movement fault is identified. Compared to existing technologies, this application, in the event of a sudden change in the image, does not directly determine a monitoring equipment fault. Instead, it further introduces a query for position adjustment commands to eliminate sudden changes in the image caused by normal position adjustments of the monitoring equipment, thus improving the accuracy of fault diagnosis. Furthermore, by comparing the theoretical image path and the actual image path, it further determines whether a fault exists in the monitoring equipment during the position adjustment process. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a security video surveillance fault detection method according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a stitched image provided in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of a theoretical image path provided in an embodiment of the present invention;

[0058] Figure 4This is a schematic diagram of an actual image path provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the actual image path provided in another embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the differential path points provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the structure of a security video surveillance fault detection device provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0064] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relativity or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0066] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0067] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0068] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0069] Please refer to Figure 1 The following is a flowchart illustrating a security video surveillance fault detection method according to an embodiment of the present invention, comprising the following specific steps:

[0070] S1. Acquire the first video frame collected by the monitoring device at the previous detection time and the second video frame collected at the current detection time; if it is determined that the first similarity between the first video frame and the second video frame is less than the first threshold, detect whether the monitoring device receives a position adjustment instruction during the interval between the previous detection time and the current detection time.

[0071] Specifically, in this invention, each detection time is preset according to a pre-defined time period. For example, a detection can be performed every minute, with a 1-minute interval between two adjacent detection times. After acquiring the video frame (the second video frame) of the monitoring device at the current detection time, the video frame acquired at the current detection time is compared with the video frame (the first video frame) acquired by the monitoring device at the previous detection time. The similarity between the two is calculated. If the similarity is less than a set first threshold, it is determined that a sudden change has occurred in the monitoring screen. This sudden change may be due to a malfunction, or it may be due to the monitoring device receiving a position adjustment command and adjusting its own position according to the adjustment method and range specified in the command, resulting in a change in the monitoring area and thus a sudden change in the screen. To distinguish between these two situations, if the similarity is less than the set first threshold, it is checked whether the monitoring device received a position adjustment command during the interval between the previous detection time and the current detection time. If so, it indicates that the sudden change in the screen is due to the monitoring device adjusting its own position; otherwise, it indicates that the sudden change in the screen is due to a malfunction.

[0072] To more accurately compare the similarity between the first video frame and the second video frame, in a preferred embodiment, the similarity between the first video frame and the second video frame can be calculated in the following way:

[0073] Determine the static image region and dynamic image region of the first video frame;

[0074] A three-dimensional coordinate system is constructed with the row coordinates of the pixel in the image as the X-axis, the column coordinates of the pixel in the image as the Y-axis, and the pixel value of the pixel as the Y-axis.

[0075] The pixels in the static image area of ​​the first video frame are projected into a three-dimensional coordinate system to generate the first spatial projection.

[0076] The pixels in the dynamic image region of the first video frame are projected into a three-dimensional coordinate system to generate a second spatial projection.

[0077] Based on the static image area of ​​the first video frame, the corresponding pixels in the second video frame are projected into the three-dimensional coordinate system to generate a third spatial projection.

[0078] Based on the dynamic image region of the first video frame, the corresponding pixels in the second video frame are projected into the three-dimensional coordinate system to generate the fourth spatial projection.

[0079] Calculate the similarity between the first spatial projection and the third spatial projection to obtain the static similarity; calculate the similarity between the second spatial projection and the fourth spatial projection to obtain the dynamic similarity.

[0080] A comprehensive similarity is calculated based on the static similarity, the first preset weight corresponding to the static similarity, the dynamic similarity, and the second preset weight corresponding to the dynamic similarity; wherein, the first preset weight is greater than the second preset weight.

[0081] The overall similarity is used as the similarity between the first video frame and the second video frame.

[0082] In real-world monitoring scenarios, a surveillance image may contain static, unchanging scenes, such as buildings and fixed equipment, or dynamically changing scenes, such as the sky. Different weather conditions cause changes in the corresponding areas of the sky, such as cloud movement or a change from sunny to cloudy. The pixel values ​​of pixels in static scenes generally do not change, but the pixel values ​​of pixels in dynamic scenes change over time. These changes reduce the similarity between two images. However, in actual monitoring, if the difference between two consecutive detection times is only in the dynamic area, we should generally consider this a normal image change and not an abrupt change. Therefore, in actual similarity comparison, it is necessary to minimize the impact of dynamic area changes on the image similarity between two consecutive detection times.

[0083] Therefore, in this embodiment of the invention, the static image region and the dynamic image region of the first video frame are first determined;

[0084] Then, the pixels of the static image region of the first video frame are projected onto the corresponding three-dimensional coordinate system according to the coordinates and pixel values ​​of the pixels in the image to generate the first spatial projection.

[0085] Next, the pixels corresponding to the static image area of ​​the first video frame are extracted in the second video frame. The extracted pixels are then projected into the corresponding three-dimensional coordinate system in the same way to generate a third spatial projection.

[0086] Then, by calculating the overlap between the first spatial projection and the third spatial projection, the similarity between the first spatial projection and the third spatial projection can be obtained, thus obtaining the static similarity between the two video frames.

[0087] Then, in a similar manner, the dynamic similarity between the two video frames is calculated;

[0088] Finally, a comprehensive similarity is calculated based on the static similarity, the first preset weight corresponding to the static similarity, the dynamic similarity, and the second preset weight corresponding to the dynamic similarity. This comprehensive similarity is used as the similarity between the first video frame and the second video frame. The illustrative calculation formula is as follows:

[0089] A = b * B + d * D; where b is the first preset weight, B is the static similarity, d is the second preset weight, and D is the dynamic similarity; b > d; b + d = 1;

[0090] By increasing the weight of static similarity and decreasing the weight of dynamic similarity, the impact of dynamic regions on the similarity between two video frames is weakened, thus better reflecting actual monitoring scenarios and improving the accuracy of fault detection in monitoring equipment.

[0091] In a preferred embodiment, determining the static image region and dynamic image region of the first video frame includes:

[0092] Based on the first standard image corresponding to the first video frame in the stitched image, the monitoring position of the monitoring device at the previous detection time is determined, and the target monitoring position is obtained.

[0093] Acquire historical video frames collected by the monitoring equipment at the target monitoring location at each historical moment; generate historical video groups based on each historical video frame; wherein each historical video group contains two historical video frames that are temporally adjacent.

[0094] For each historical video group, each historical video frame within the historical video group is divided into several tiles. The second similarity between the tiles corresponding to two historical video frames is calculated one by one. The region corresponding to the tile group with the second similarity less than the second threshold is taken as the dynamic region to be selected for the historical video group.

[0095] The intersection of the selected dynamic regions corresponding to all historical video groups is used to obtain the dynamic image region of the first video frame.

[0096] All image regions in the first video frame, excluding dynamic image regions, are considered as static image regions.

[0097] First, we need to explain the stitched image:

[0098] like Figure 2 As shown, in this invention, under normal circumstances, the monitoring equipment is controlled to move vertically and horizontally and capture video frames corresponding to each monitoring position. These video frames are then stitched together according to their positions to form a... Figure 2 The stitched image shown; in Figure 2In the stitched image shown, assuming video frame A is the initial position, the video frames in each row represent the video frames captured when the monitoring device moves horizontally, and the video frames in each column represent the video frames captured when the monitoring device moves vertically. The monitoring positions corresponding to adjacent video frames are adjacent. The entire stitched image represents the view of all monitored areas that the monitoring device can cover during a historical period, assuming no faults.

[0099] When identifying the static and dynamic image regions of the first video frame, the similarity between the first video frame and each video frame in the stitched image is calculated. Existing similarity calculation methods can be used for this calculation. For example, the pixel values ​​of all pixels in the first video frame are extracted, and then the pixel values ​​of all pixels in a video frame to be compared in the stitched image are extracted. The proportion of pixels with the same pixel value at corresponding positions between the first video frame and the video frame to be compared is then calculated and used as the similarity between the first video frame and the video frame to be compared in the stitched image. Using this method, the image with the highest similarity to the first video frame is selected from the stitched image to obtain the first standard image corresponding to the first video frame in the stitched image. Next, based on the first standard image corresponding to the first video frame in the stitched image, its corresponding monitoring position is determined, resulting in the target monitoring position. This monitoring position is the location where the monitoring equipment acquired the first video frame.

[0100] Next, based on the historical monitoring data corresponding to the target monitoring location, the historical video frames collected by the monitoring equipment at the target monitoring location at each historical moment are obtained; then, the historical video groups are divided so that each historical video group contains two historical video frames that are temporally adjacent.

[0101] For each historical video group, each historical video frame within the group is divided into several tiles. This division can be based on the actual size of the video frame and a preset number of tiles (e.g., 64 tiles). Then, the similarity between corresponding tiles of two historical video frames is calculated one by one. This similarity calculation also uses existing methods; for example, the pixel values ​​of all pixels in one tile are extracted, and then the pixel values ​​of all pixels in another tile are extracted. The percentage of pixels with the same pixel value at corresponding positions in the two tiles is then counted as the similarity between the two tiles, resulting in the second similarity. The regions corresponding to tile groups with a second similarity less than a second threshold are designated as the candidate dynamic regions for the historical video group. This step allows for the selection of regions in each historical video group where pixel changes occur between consecutive video frames. Since the monitoring equipment is located at the same location, the pixel changes are largely due to dynamic scenes occurring in the area corresponding to the monitoring location.

[0102] Then, the intersection of the candidate dynamic regions corresponding to the historical video group is taken to obtain the dynamic image region of the first video frame. By taking the intersection of the candidate dynamic regions of the historical video group, the influence of some occasional events on the image changes can be eliminated. For example, if a dog suddenly appears in the monitored area at a certain historical moment and then quickly disappears, the dog will only appear in a few historical video frames. By taking the intersection of the candidate dynamic regions of the historical video group, the area in the monitored area that is in a long-term dynamic change can be accurately identified, and thus the dynamic image region of the first video frame can be determined. Finally, all image regions in the first video frame other than the dynamic image region are taken as static image regions, thus completing the division of static image regions and dynamic image regions in the first video frame.

[0103] S2. If not, then the monitoring equipment is confirmed to have an image acquisition failure.

[0104] If so, the theoretical monitoring positions that the monitoring equipment should be set at each time point in the interval period are determined according to the position adjustment command; the video frames at the corresponding positions are identified from the preset stitched images according to each theoretical monitoring position to obtain each first standard video frame; the center point of each first standard video frame is extracted, and the center points of each first standard video frame are connected in chronological order according to the time corresponding to the theoretical monitoring position to generate the theoretical image path that the monitoring equipment should correspond to when it moves according to the position adjustment command; wherein, the video frames captured by the monitoring equipment at all monitoring positions in the historical period are stitched together according to the corresponding monitoring positions to generate a stitched image.

[0105] Specifically, if the monitoring equipment does not receive a position adjustment command within the interval period, it indicates that the sudden change in the screen is caused by a malfunction, and the monitoring equipment is judged to have an image acquisition failure.

[0106] To further determine the type and cause of the image acquisition failure, the following methods can be used for further identification:

[0107] In a preferred embodiment, if it is determined that the monitoring device has an image acquisition failure, the method further includes: extracting the RGB values ​​of each pixel in the second video frame; if the RGB values ​​of all pixels are all zero, it is determined that the monitoring device has a black screen failure; if the RGB values ​​of all pixels are all 255, it is determined that the monitoring device has a white screen failure; otherwise, it is determined that the monitoring device has been illegally moved or has been subjected to an image tampering attack.

[0108] In a preferred embodiment, after determining that the monitoring device has experienced a black screen malfunction, the method further includes:

[0109] Based on each actual video frame actually collected by the monitoring equipment at each time interval, determine whether there is a person or object continuously approaching the monitoring equipment.

[0110] If so, the black screen failure is determined to be caused by malicious human obstruction; otherwise, the black screen failure is determined to be caused by damage to the monitoring equipment.

[0111] If the monitoring device received a position adjustment command within the interval period, it means that this sudden change in the screen is caused by the position adjustment of the monitoring device.

[0112] At this point, the theoretical monitoring position that the monitoring equipment should be set at each moment in the interval period is first determined according to the position adjustment command. Specifically, the monitoring position corresponding to the previous detection moment is taken as the initial monitoring position. Then, according to the adjustment direction and adjustment range (i.e., the distance moved) contained in the position adjustment command, the theoretical position that the monitoring equipment should be at each moment in the process of adjusting from the initial monitoring position to the final position according to the position adjustment command (e.g., every frame is taken as a moment), the theoretical position that the monitoring equipment should be at each moment between the initial moment (i.e., the previous detection moment) and the moment of receiving the position adjustment command, and the theoretical position that the monitoring equipment should be at each moment between the moment when the monitoring equipment adjusts to the final position and the termination moment (i.e., the current detection moment) are obtained. Thus, the theoretical monitoring position that the monitoring equipment should be set at each moment in the interval period is obtained.

[0113] Next, based on the theoretical monitoring positions, the corresponding video frames are identified from the pre-set stitched images. (Illustrative example follows.) Figure 3As shown, assuming that the monitoring device was at the position corresponding to video frame A of the stitched image at the previous detection time, it received a position adjustment command at the previous detection time. The received position adjustment command instructed the monitoring device to first move horizontally to the right by two units, and then move vertically upward by three units, and the time of adjustment completion coincided with the current detection time. Then, the theoretical monitoring positions corresponding to each time interval at this time correspond to the first standard video frames in the stitched image as video frames A, B, C, D, E, and F, respectively. Then, the image center points of video frames A, B, C, D, E, and F are connected in sequence to obtain the theoretical image path L1 mentioned above.

[0114] S3. Calculate the similarity between each actual video frame collected by the monitoring equipment and each video frame in the stitched image at each time point during the interval period. Take the video frame with the highest similarity in the stitched image as the second standard video frame corresponding to the actual video frame. Extract the center point of each second standard video frame and connect the center points of each second standard video frame in chronological order to generate the actual image path corresponding to the actual movement of the monitoring equipment.

[0115] Specifically, after determining the theoretical image path L1, each actual video frame captured by the monitoring equipment at each time interval is obtained. These actual video frames represent the video frames actually captured by the monitoring equipment at each time interval. If the monitoring equipment correctly adjusts its position according to the position adjustment command, the actual movement trajectory should be consistent with the theoretical movement trajectory. Correspondingly, in the stitched image, the theoretical image path should be consistent with the actual image path. Based on this principle, for each actual video frame, the similarity between the actual video frame and each video frame in the stitched image is calculated. The video frame with the highest similarity in the stitched image is taken as the second standard video frame corresponding to the actual video frame. This allows the correspondence between the images captured by the monitoring equipment during its actual movement and the video frames in the stitched image to be determined, thus identifying the monitoring location points traversed by the monitoring equipment during its actual movement without other positioning devices. Then, the image center points of each second standard video frame are connected to generate the actual image path. It should be noted that when calculating the similarity between the actual video frame and each video frame in the stitched image, the existing method is also used. For example: extract the pixel values ​​of all pixels in the actual video frame, then extract the pixel values ​​of all pixels in a video frame in the stitched image, and then count the proportion of pixels with the same pixel value at corresponding positions between the actual video frame and a video frame in the stitched image, and use this proportion as the similarity between the actual video frame and a video frame in the stitched image.

[0116] Indicative, such as Figure 4As shown, assuming the monitoring device accurately adjusts its position according to the position adjustment command, the corresponding second standard video frames are as follows: video frame A, video frame B, video frame C, video frame D, video frame E, and video frame F. Then, the center points of the images of video frames A, B, C, D, E, and F are connected in sequence to obtain the theoretical image path L2 mentioned above.

[0117] Indicative, such as Figure 5 As shown, assuming the monitoring device does not correctly adjust its position according to the position adjustment command, it first moves horizontally to the right by two units, and then moves vertically downwards by three units. The corresponding second standard video frames at this time are, in order: video frame A, video frame B, video frame C, video frame G, video frame H, and video frame I. Then, the center points of the images of video frames A, B, C, G, H, and I are connected in sequence to obtain the theoretical image path L2 mentioned above.

[0118] S4. Determine whether the theoretical image path and the actual image path are consistent. If they are consistent, it is determined that the monitoring equipment is not faulty. If they are inconsistent, it is determined that the monitoring equipment has a movement fault.

[0119] Specifically, the stitched images containing the theoretical image path (such as...) Figure 3 ) and stitched images with actual image paths (such as Figure 4 After overlapping, compare whether the theoretical image path and the actual image path completely overlap. If they do, it is determined that the theoretical image path and the actual image path are consistent. This indicates that the monitoring equipment has moved accurately according to the position adjustment command, and the second video frame acquired at the current detection moment is also correct, meaning that the monitoring equipment has not malfunctioned. Otherwise, it is determined that the theoretical image path and the actual image path are inconsistent, indicating that the monitoring equipment has a movement malfunction.

[0120] In a preferred embodiment, when it is determined that the monitoring device has a movement fault, the path point where the difference between the theoretical image path and the actual image path begins to occur is determined based on the overlap between the theoretical image path and the actual image path, and the difference path point is obtained.

[0121] Based on the time and location corresponding to the video frame where the difference path point is located, determine the time and location when the monitoring equipment experiences a movement failure.

[0122] Specifically, assuming the generated actual image path is Figure 5 As shown, Figure 5 and Figure 3 After comparison, the path points of difference can be determined. Figure 6 The center point of video frame C;

[0123] Based on the time and location corresponding to video frame C, the specific time and location when the monitoring equipment experienced a movement failure can be determined.

[0124] In a preferred embodiment, after determining that the monitoring device has a mobility fault, the method further includes:

[0125] Determine the positions of the first standard video frames corresponding to the previous detection time and the current detection time in the stitched image, and obtain the first position and the second position respectively;

[0126] Based on the first position and the second position, in the stitched image, when determining the image path from the first position to the second position that avoids the difference path points and has the shortest path, the target image path is obtained;

[0127] The monitoring equipment is controlled to move sequentially according to the monitoring positions corresponding to the target image path to complete the position correction.

[0128] In this embodiment, position correction is required after a movement fault occurs. At this time, the position of the first standard video frame corresponding to the previous detection time and the current detection time in the stitched image can be determined first to obtain the initial position (i.e., the first position) and the final position (i.e., the second position) when the position adjustment instruction is performed.

[0129] Next, using an enumeration method, all image paths from the first position to the second position are traversed on the stitched image. The shortest path, without any discrepancy points, is then selected to obtain the target image path. Finally, the monitoring device is controlled to move sequentially according to the monitoring positions corresponding to the target image path, completing the position correction. This position correction method avoids discrepancy points. Since the theoretical and actual image paths differ at these discrepancy points, it indicates that there may be obstacles at the monitoring position of the video frame corresponding to the discrepancy point, preventing further adjustment. Therefore, selecting a target image path that avoids the discrepancy point during position correction improves the success rate.

[0130] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments;

[0131] Please refer to Figure 7 A schematic diagram of the structure of a security video surveillance fault detection device provided in an embodiment of the present invention is shown. The device includes: an instruction determination module, a first fault identification module, a theoretical image path generation module, an actual image path generation module, and a second fault identification module.

[0132] The instruction determination module is used to acquire the first video frame collected by the monitoring device at the previous detection time and the second video frame collected at the current detection time; if it is determined that the first similarity between the first video frame and the second video frame is less than the first threshold, it detects whether the monitoring device receives a position adjustment instruction during the interval between the previous detection time and the current detection time.

[0133] The first fault identification module is used to determine that the monitoring equipment has an image acquisition fault when it is determined that the monitoring equipment has not received a position adjustment instruction;

[0134] The theoretical image path generation module is used to determine the theoretical monitoring positions that the monitoring equipment should set at each time point in the interval period when the monitoring equipment receives a position adjustment instruction; to identify the video frames at the corresponding positions from the preset stitched images according to each theoretical monitoring position, to obtain each first standard video frame, to extract the center point of each first standard video frame, and to connect the center points of each first standard video frame in chronological order according to the time corresponding to the theoretical monitoring position, thereby generating the theoretical image path that the monitoring equipment should correspond to when it moves according to the position adjustment instruction; wherein, the video frames captured by the monitoring equipment at all monitoring positions in the historical period are stitched together according to the corresponding monitoring positions to generate a stitched image.

[0135] The actual image path generation module is used to calculate the similarity between each actual video frame actually collected by the monitoring device and each video frame in the stitched image at each time of the interval period. The video frame with the highest similarity in the stitched image is used as the second standard video frame corresponding to the actual video frame. The center point of each second standard video frame is extracted, and the center points of each second standard video frame are connected in chronological order to generate the actual image path corresponding to the actual movement of the monitoring device.

[0136] The second fault identification module is used to determine whether the theoretical image path and the actual image path are consistent. If they are consistent, it is determined that the monitoring equipment has not malfunctioned. If they are inconsistent, it is determined that the monitoring equipment has a movement fault.

[0137] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0138] Those skilled in the art will clearly understand that, for convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0139] Accordingly, embodiments of the present invention provide an electronic device, the device including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the security video surveillance fault detection method described in the above embodiments of the invention.

[0140] The electronic device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The device may include, but is not limited to, a processor and a memory.

[0141] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the device, connecting various parts of the device via various interfaces and lines.

[0142] Accordingly, embodiments of the present invention provide a storage medium, the storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the storage medium is located to execute the security video surveillance fault detection method described in the above embodiments of the invention.

[0143] The memory can be used to store the computer program. The processor implements various functions of the device by running or executing the computer program stored in the memory and calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0144] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0145] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting faults in security video surveillance, characterized in that, include: The system acquires a first video frame captured by the monitoring device at the previous detection time and a second video frame captured at the current detection time; if the first similarity between the first video frame and the second video frame is less than a first threshold, the system detects whether the monitoring device receives a position adjustment command during the interval between the previous detection time and the current detection time. If not, then the monitoring equipment is confirmed to have an image acquisition failure; If so, the theoretical monitoring positions that the monitoring equipment should be set at each time point in the interval period are determined according to the position adjustment command; the video frames at the corresponding positions are identified from the preset stitched images according to each theoretical monitoring position to obtain each first standard video frame; the center point of each first standard video frame is extracted, and the center points of each first standard video frame are connected in chronological order according to the time corresponding to the theoretical monitoring position to generate the theoretical image path that the monitoring equipment should correspond to when it moves according to the position adjustment command; wherein, the video frames captured by the monitoring equipment at all monitoring positions in the historical period are stitched together according to the corresponding monitoring positions to generate a stitched image. Calculate the similarity between each actual video frame collected by the monitoring equipment and each video frame in the stitched image at each time point during the interval period. Take the video frame with the highest similarity in the stitched image as the second standard video frame corresponding to the actual video frame. Extract the center point of each second standard video frame and connect the center points of each second standard video frame in chronological order to generate the actual image path corresponding to the actual movement of the monitoring equipment. Determine whether the theoretical image path and the actual image path are consistent. If they are consistent, it is determined that the monitoring equipment is not faulty; if they are inconsistent, it is determined that the monitoring equipment has a movement fault.

2. The security video surveillance fault detection method as described in claim 1, characterized in that, The first similarity between the first video frame and the second video frame is calculated in the following manner: Determine the static image region and dynamic image region of the first video frame; A three-dimensional coordinate system is constructed with the row coordinates of the pixel in the image as the X-axis, the column coordinates of the pixel in the image as the Y-axis, and the pixel value of the pixel as the Y-axis. The pixels in the static image area of ​​the first video frame are projected into a three-dimensional coordinate system to generate the first spatial projection. The pixels in the dynamic image region of the first video frame are projected into a three-dimensional coordinate system to generate a second spatial projection. Based on the static image area of ​​the first video frame, the corresponding pixels in the second video frame are projected into the three-dimensional coordinate system to generate a third spatial projection. Based on the dynamic image region of the first video frame, the corresponding pixels in the second video frame are projected into the three-dimensional coordinate system to generate the fourth spatial projection. Calculate the similarity between the first spatial projection and the third spatial projection to obtain the static similarity; Calculate the similarity between the second spatial projection and the fourth spatial projection to obtain the dynamic similarity; A comprehensive similarity is calculated based on the static similarity, the first preset weight corresponding to the static similarity, the dynamic similarity, and the second preset weight corresponding to the dynamic similarity; wherein, the first preset weight is greater than the second preset weight. The overall similarity is used as the first similarity between the first video frame and the second video frame.

3. The security video surveillance fault detection method as described in claim 2, characterized in that, Determining the static image region and dynamic image region of the first video frame includes: Based on the first standard image corresponding to the first video frame in the stitched image, the monitoring position of the monitoring device at the previous detection time is determined, and the target monitoring position is obtained. Acquire historical video frames collected by the monitoring equipment at the target monitoring location at each historical moment; generate historical video groups based on each historical video frame; wherein each historical video group contains two historical video frames that are temporally adjacent. For each historical video group, each historical video frame within the historical video group is divided into several tiles. The second similarity between the tiles corresponding to two historical video frames is calculated one by one. The region corresponding to the tile group with the second similarity less than the second threshold is taken as the dynamic region to be selected for the historical video group. The intersection of the selected dynamic regions corresponding to all historical video groups is used to obtain the dynamic image region of the first video frame. All image regions in the first video frame, excluding dynamic image regions, are considered as static image regions.

4. The security video surveillance fault detection method as described in claim 3, characterized in that, Also includes: If it is determined that the monitoring device has a movement fault, the path point where the difference between the theoretical image path and the actual image path begins to appear is determined based on the overlap between the theoretical image path and the actual image path, and the difference path point is obtained. Based on the time and location corresponding to the video frame where the difference path point is located, determine the time and location when the monitoring equipment experiences a movement failure.

5. The security video surveillance fault detection method as described in claim 4, characterized in that, After determining that the monitoring device has a mobility fault, the process also includes: Determine the positions of the first standard video frames corresponding to the previous detection time and the current detection time in the stitched image, and obtain the first position and the second position respectively; Based on the first position and the second position, in the stitched image, when determining the image path from the first position to the second position that avoids the difference path points and has the shortest path, the target image path is obtained; The monitoring equipment is controlled to move sequentially according to the monitoring positions corresponding to the target image path to complete the position correction.

6. The security video surveillance fault detection method as described in claim 5, characterized in that, In cases where it is determined that the monitoring equipment has experienced an image acquisition failure, the following additional steps are also included: Extract the RGB values ​​of each pixel in the second video frame; If the RGB values ​​of all pixels are all zero, then the monitoring device is confirmed to have a black screen fault. If the RGB value of all pixels is 255, then the monitoring device is confirmed to have a white screen fault. Otherwise, it could indicate that the monitoring equipment has been illegally moved or subjected to an image tampering attack.

7. The security video surveillance fault detection method as described in claim 6, characterized in that, After confirming that the monitoring equipment is experiencing a black screen malfunction, the following steps are also included: Based on each actual video frame actually collected by the monitoring equipment at each time interval, determine whether there is a person or object continuously approaching the monitoring equipment. If so, the black screen failure is determined to be caused by malicious human obstruction; otherwise, the black screen failure is determined to be caused by damage to the monitoring equipment.

8. A security video surveillance fault detection device, characterized in that, include: The system comprises an instruction determination module, a first fault identification module, a theoretical image path generation module, an actual image path generation module, and a second fault identification module. The instruction determination module is used to acquire the first video frame collected by the monitoring device at the previous detection time and the second video frame collected at the current detection time; if it is determined that the first similarity between the first video frame and the second video frame is less than the first threshold, it detects whether the monitoring device receives a position adjustment instruction during the interval between the previous detection time and the current detection time. The first fault identification module is used to determine that the monitoring equipment has an image acquisition fault when it is determined that the monitoring equipment has not received a position adjustment instruction; The theoretical image path generation module is used to determine the theoretical monitoring positions that the monitoring equipment should set at each time point in the interval period when the monitoring equipment receives a position adjustment instruction; to identify the video frames at the corresponding positions from the preset stitched images according to each theoretical monitoring position, to obtain each first standard video frame, to extract the center point of each first standard video frame, and to connect the center points of each first standard video frame in chronological order according to the time corresponding to the theoretical monitoring position, thereby generating the theoretical image path that the monitoring equipment should correspond to when it moves according to the position adjustment instruction; wherein, the video frames captured by the monitoring equipment at all monitoring positions in the historical period are stitched together according to the corresponding monitoring positions to generate a stitched image. The actual image path generation module is used to calculate the similarity between each actual video frame actually collected by the monitoring device and each video frame in the stitched image at each time of the interval period. The video frame with the highest similarity in the stitched image is used as the second standard video frame corresponding to the actual video frame. The center point of each second standard video frame is extracted, and the center points of each second standard video frame are connected in chronological order to generate the actual image path corresponding to the actual movement of the monitoring device. The second fault identification module is used to determine whether the theoretical image path and the actual image path are consistent. If they are consistent, it is determined that the monitoring equipment has not malfunctioned; if they are inconsistent, it is determined that the monitoring equipment has a movement fault.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the security video surveillance fault detection method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the storage medium is located to perform the security video surveillance fault detection method as described in any one of claims 1 to 7.

Images