A missed video frame detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a kind of detection method, device, electronic equipment and storage medium of missed video frame, wherein method includes: according to the motion target detected from monitoring video frame, determine missed video frame;According to the position parameter of motion target in the previous frame and the next frame of the missed video frame, demarcate missed area in the missed video frame;Missed area is verified based on the motion region detected from the missed video frame.The present disclosure scheme automatically determines missed video frame, and demarcates the missed area of missed video frame, avoids manual demarcation, improves the efficiency of missed video frame demarcation;And use the motion region of missed video frame, verify whether the demarcated missed area is accurate, guarantee the accuracy of the missed area finally obtained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the security field, and in particular to a missed video frame detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the gradual intelligence of network cameras, intelligent modules are usually embedded, and intelligent alarm can be realized through algorithm analysis. The current algorithm index is often strongly dependent on strict installation according to the reconnaissance scene. However, in actual use scenarios, customer environments vary greatly, and it is often difficult to strictly implement reconnaissance requirements, so that the detected target may be missed due to changes in angle and size, resulting in discontinuous live box, multiple reports of the same target, and the like.

[0003] Currently, in order to reduce the probability of picture missing, the algorithm in the customer environment needs to be optimized. The specific optimization process is: the customer feedback including the missing picture video is calibrated by manual, and then the missing picture is calibrated by manual. However, this way not only needs manual to determine the missing picture from the video, but also is very labor-intensive and low in calibration efficiency. SUMMARY

[0004] The present disclosure provides a missed video frame detection method, device, electronic equipment and storage medium to improve the calibration efficiency of the missing picture.

[0005] According to an aspect of the present disclosure, a missed video frame detection method is provided, comprising:

[0006] According to the motion target detected from the monitoring video frame, a missed video frame is determined;

[0007] According to the position parameters of the motion target in the previous frame and the next frame of the missed video frame, a missing area is calibrated in the missed video frame;

[0008] Based on the motion area detected from the missed video frame, the missing area is verified.

[0009] According to another aspect of the present disclosure, a missed video frame detection device is provided, comprising:

[0010] The missed frame determination module is configured to determine a missed video frame according to the motion target detected from the monitoring video frame;

[0011] The calibration module is configured to calibrate a missing area in the missed video frame according to the position parameters of the motion target in the previous frame and the next frame of the missed video frame;

[0012] A verification module is configured to verify the missed region based on the motion region detected from the missed video frame.

[0013] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory in communication with the at least one processor; wherein

[0016] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for detecting a missed video frame according to any of the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, and the computer instructions are used to enable a computer to perform the method for detecting a missed video frame according to any of the embodiments of the present disclosure.

[0018] According to the technology of the present disclosure, a missed video frame is automatically determined, and a missed region of the missed video frame is labeled, which avoids manual labeling and improves the efficiency of labeling the missed video frame.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0021] Figure 1a is a flowchart of a method for detecting a missed video frame according to an embodiment of the present disclosure;

[0022] Figure 1b is a schematic diagram of a continuous video frame including a moving target according to an embodiment of the present disclosure;

[0023] Figure 1c is another schematic diagram of a continuous video frame including a moving target according to an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart of another method for detecting a missed video frame according to an embodiment of the present disclosure;

[0025] Figure 3a is a flowchart of another method for detecting a missed video frame according to an embodiment of the present disclosure;

[0026] Figure 3bis a schematic diagram of a continuous video including a moving target provided by an embodiment of the present disclosure;

[0027] Figure 4a is a flowchart of a detection method of a missed video frame provided by an embodiment of the present disclosure;

[0028] Figure 4b is a schematic diagram of a continuous video frame including a moving target provided by an embodiment of the present disclosure;

[0029] Figure 5a is a flowchart of a detection method of a missed video frame provided by an embodiment of the present disclosure;

[0030] Figure 5b is a schematic diagram of a continuous video frame including a missed video frame provided by an embodiment of the present disclosure;

[0031] Figure 6 is a flowchart of a detection method of a missed video frame provided by an embodiment of the present disclosure;

[0032] Figure 7 is a flowchart of a detection method of a missed video frame provided by an embodiment of the present disclosure;

[0033] Figure 8 is a structural schematic diagram of a detection apparatus of a missed video frame provided by an embodiment of the present disclosure;

[0034] Figure 9 is a block diagram of an electronic device for implementing a detection method of a missed video frame according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help in understanding. These should be considered in the context of the overall description and they should not be taken in a limiting sense. Thus, one of ordinary skill in the art will understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0036] The embodiments of the present disclosure are suitable for monitoring scenes, in which at least one network camera is arranged, and the network camera can detect moving targets in the scene based on a pre-deployed target detection algorithm model, so as to achieve the purposes of target tracking or intrusion detection. However, in some monitoring scenes, the algorithm may be missed detection due to reasons such as brightness of the monitoring area, shooting angle of the camera, and partial or complete occlusion. At this time, it can be determined that the target detection algorithm model in the network camera is not completely suitable for the scene, and the target detection algorithm model needs to be optimized. A common optimization method is to manually calibrate the scene materials (such as videos including missed detection video frames) provided by the user, and then train the target detection algorithm model using the calibrated images. However, in this model optimization method, the user needs to provide materials and manual calibration, which results in low calibration efficiency and wastes a lot of manpower and time. Therefore, a missed detection video frame detection method is proposed, which automatically determines the missed detection video frame using the network camera and calibrates the missed detection video frame, so as to improve the efficiency of calibrating the missed detection video frame. For the specific implementation process, refer to the following embodiments.

[0037] Figure 1a For a flowchart of a missed detection video frame detection method of the embodiments of the present disclosure, the present embodiment can be suitable for automatically determining the missed detection video frame using the network camera and calibrating the missed detection video frame. The method can be executed by a missed detection video frame detection device, which is realized in the form of software and / or hardware and integrated on an electronic device, such as a network camera.

[0038] Specifically, referring to Figure 1a , the flow of the missed detection video frame detection method is as follows:

[0039] S101, determining a missed detection video frame according to a moving target detected from a monitoring video frame.

[0040] The motion target in the monitoring scene can be selected from at least one of a pedestrian, a motor vehicle, or a non-motor vehicle, and the same motion target is marked by the same target identification code (for example, a target ID). In the embodiments of the present disclosure, the essence of the missed detection is to detect a certain historical motion target as a new motion target, so that the historical motion target and the new motion target are completely the same except for the target identification code. Therefore, if it is detected that two motion targets are completely the same except for the target identification code in the target detection process, it is determined that the missed detection occurs in the target detection process, and at this time, the missed video frame needs to be determined. It should be noted that the missed video frame refers to a video frame in which a motion target is not detected due to factors such as light factors (luminance difference of part of the video region), shadow projection, physical occlusion, and the angle of part of the video region relative to the camera does not meet the installation requirements. For example, for the case of physical occlusion, the missed video frame can be a video frame including a partially occluded motion target, or a video frame in which the motion target is completely occluded (i.e., a video frame in which there is no motion target in the picture).

[0041] In S102, the missed detection region is marked in the missed video frame according to the position parameters of the motion target in the previous frame and the next frame of the missed video frame.

[0042] In the embodiments of the present disclosure, the number of missed video frames can be one frame or multiple frames. If the number of missed video frames is one frame, the previous frame of the missed video frame refers to the video frame collected at the moment before the missed video frame is collected, and the next frame of the missed video frame refers to the video frame collected at the moment after the missed video frame is collected. If the number of missed video frames is multiple frames, according to the order from the earliest to the latest according to the collection time of each missed video frame, the previous frame of the missed video frame refers to the video frame collected at the moment before the earliest missed video frame is collected, and the next frame of the missed video frame refers to the video frame collected at the moment after the latest missed video frame is collected.

[0043] For example, referring to Figure 1b and Figure 1c , which show a schematic diagram of continuous video frames including a motion target, wherein each video frame has a unique frame identification code (for example, a frame number). For example, in Figure 1b , if it is determined through S101 that the video frame with frame number 00003 is a missed video frame, the previous frame of the missed video frame refers to the video frame with frame number 00002, and the next frame of the missed video frame refers to the video frame with frame number 00004. In Figure 1c , if it is determined through S101 that the video frames with frame numbers 00002, 00003, and 00004 are all missed video frames, the previous frame of the missed video frame refers to the video frame with frame number 00001, and the next frame of the missed video frame refers to the video frame with frame number 00005.

[0044] In the embodiments of the present disclosure, the position parameters of the moving object in the previous frame and the next frame of the missed video frame can be position coordinates representing the range of the area occupied by the moving object. For example, the position parameters of the moving object in the previous frame and the next frame are the position coordinates of the upper left corner and the upper right corner of the moving object. According to the position parameters of the moving object in the previous frame and the next frame, the missed area in the missed video frame is predicted by using a preset prediction formula. The missed area in the missed video frame can also be predicted by other methods, which are not limited here. It should be noted that in addition to the position parameters of the missed area, the type of the moving object included in the missed area can also be determined. Since the type of the missed video frame is consistent with the type of the moving object in the previous frame and the next frame, the type of the moving object can be determined from the target detection result of the previous frame or the next frame, and the type of the moving object in the missed video frame can be determined.

[0045] S103, verifying the missed area based on the moving area detected from the missed video frame.

[0046] In the embodiments of the present disclosure, if the missed area is determined according to the steps of S101 and S102 when the moving object enters a completely occluded environment and then drives out of the completely occluded environment, there is a possibility that the image area of the environment is mistaken for the missed area. Therefore, it is necessary to verify this situation. Optionally, whether the determined missed area is accurate is verified according to whether there is a moving area in the missed video frame and the relationship between the moving area and the missed area, wherein the moving area is obtained by detecting the missed video frame based on a motion detection algorithm. For example, if there is no moving area in the missed video frame, it is considered that the determined missed area is incorrect, that is, the determined missed area is actually the image of the environment; if there is a moving area in the missed video frame, and the moving area partially overlaps with the missed area, it is determined that the determined missed area is accurate.

[0047] In the embodiments of the present disclosure, the network camera automatically determines the missed video frame and simultaneously determines the missed area of the missed video frame, thereby avoiding manual determination and improving the efficiency of the determination of the missed video frame. Moreover, whether the determined missed area is accurate is verified by using the moving area of the missed video frame, thereby ensuring the accuracy of the finally obtained missed area.

[0048] Figure 2 is a flowchart of another method for detecting a missed video frame according to the embodiments of the present disclosure. The present embodiment optimizes the operation of determining the missed video frame on the basis of the above-mentioned embodiments. Referring to Figure 2 , the specific process of the method for detecting a missed video frame is as follows:

[0049] S201. Determine the missed video frames based on the attribute parameters or motion parameters of the moving targets in the monitored video frames.

[0050] In the monitoring scenario of this disclosure embodiment, the moving target can be at least one of pedestrians, motor vehicles, or non-motor vehicles, and the same moving target is labeled with the same target identification code (e.g., target ID). Attribute parameters are used to characterize the features of the moving target. For example, if the moving target is a motor vehicle, the attribute parameters of the motor vehicle include at least the vehicle type, color, license plate number, etc. Motion parameters are used to characterize the motion state of the moving target in the monitoring scenario. For example, motion parameters include at least one of the following: the size of the moving target in the video frame, the displacement vector of the moving target, and the position of the moving target.

[0051] In this embodiment, the system can determine whether two moving targets with different target identifiers are the same target based on the attribute parameters or motion parameters of the moving targets in the monitored video frames, thereby determining whether a missed detection has occurred. For example, if two moving targets with different target identifiers have completely identical attribute parameters, they are considered to belong to the same target, meaning a missed detection has occurred during the detection process. Similarly, if a historical moving target disappears from the video frame during target detection, and a new moving target appears in a subsequent video frame, and the motion parameters of the two moving targets determine that they are the same target, then a missed detection has occurred. In the case of a missed detection, the network camera can directly identify the missed video frame from the monitored video frames. A missed video frame refers to a video frame where the moving target was not detected due to factors such as lighting (brightness differences in some video areas), shadow projection, physical obstruction, or the angle of some video areas relative to the camera not conforming to the installation survey. For example, in the case of physical obstruction, the missed video frame can be either a video frame containing a partially obstructed moving target or a video frame where the moving target is completely obstructed (i.e., a video frame without a moving target).

[0052] S202. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, mark the missed detection area in the missed video frame.

[0053] S203. Verify the missed detection region based on the motion region detected from the missed video frame.

[0054] The process of calibrating and verifying the missed detection area is described in the above embodiments and will not be repeated here.

[0055] In this embodiment of the disclosure, based on the attribute parameters or motion parameters of the moving target in the monitoring video frame, it is possible to quickly and accurately detect whether a missed detection has occurred, and then determine the missed video frame from the video frame. This ensures that the missed video frame is detected in a timely manner so that subsequent calibration can be performed.

[0056] Figure 3a This is a flowchart illustrating another method for detecting missed video frames according to an embodiment of the present disclosure. This embodiment optimizes the process of determining missed video frames based on attribute parameters based on the above embodiments. See [link to relevant documentation]. Figure 3a The specific process for detecting missed video frames is as follows:

[0057] S301. In response to the detection that the target identification codes of the first moving target and the second moving target are different but the attribute parameters are the same, it is determined that a video frame has been missed.

[0058] S302. Based on the frame identifier code of the video frame including the first moving target and the frame identifier code of the video frame including the second moving target, determine the missed video frames.

[0059] In this embodiment of the disclosure, the target identifier code of the moving target can be either a target ID or other unique identifier, and no specific limitation is made here. For example, see [link to example]. Figure 3b The diagram illustrates a continuous video stream containing moving targets. A moving target with a target ID (target identifier) ​​of 00001 is designated as the first moving target, and a moving target with a target ID of 00002 is designated as the second moving target. By comparison, it can be seen that if the target identifiers of the first and second moving targets are different but their attribute parameters are the same (e.g., both are black cars), then a missed detection occurs, meaning that a video frame has been missed.

[0060] In this embodiment of the disclosure, to determine the missed video frame, it is necessary to first determine the video frames before and after the missed detection occurred, that is, to determine the last captured video frame A containing the first moving target, and the first captured video frame B containing the second moving target. Since the frame identifier code (i.e., frame number) of each video frame increases sequentially by integers, the number and frame number of the video frames between the two frames can be determined based on the frame numbers of video frame A and video frame B. For example, see [link to example]. Figure 3b Video frame A is video frame number 00002, and video frame B is video frame number 00004. Therefore, the number of missed video frames is one frame, with frame number 00003.

[0061] S303. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, mark the missed detection area in the missed video frame.

[0062] S304. Verify the missed detection region based on the motion region detected from the missed video frame.

[0063] The process of calibrating and verifying the missed detection area is described in the above embodiments and will not be repeated here.

[0064] This embodiment of the disclosure can accurately detect whether a missed detection has occurred by comparing the target identifier code and attribute parameters of a moving target. Then, when a missed detection occurs, the missed video frame can be quickly and accurately located based on the frame identifier code of the video frame.

[0065] Figure 4a This is a flowchart illustrating another method for detecting missed video frames according to an embodiment of the present disclosure. This embodiment optimizes the process of determining missed video frames based on attribute parameters based on the above embodiments. See [link to relevant documentation]. Figure 4a The specific process for detecting missed video frames is as follows:

[0066] S401. In response to the detection that a third moving target in a monitored video frame has disappeared, detect in real time whether a fourth moving target appears in the video frame.

[0067] In this embodiment of the disclosure, the third moving target and the fourth moving target refer to moving targets with different target IDs (i.e., target identification codes), and the number of fourth moving targets can be one or more, which is not limited here.

[0068] S402. After detecting the fourth moving target, determine whether the third and fourth moving targets are the same based on their motion parameters.

[0069] In this embodiment of the disclosure, the motion parameters include at least one of the following: the size of the moving target in the video frame, the displacement vector of the moving target, and the position of the moving target. Therefore, it can be determined whether the third and fourth moving targets are the same based on the size change data of the third and fourth moving targets, the angle between the displacement vectors of the third and fourth moving targets, or the positional relationship between the third and fourth moving targets.

[0070] For example, see Figure 4b The diagram illustrates a series of video frames including moving targets, where target A is the third moving target, and targets B and C are both fourth moving targets. The displacement vector of any moving target is the vector from the position where the moving target was first detected to the position where it was last detected. For example, the displacement vector of target A is vector A. To determine whether targets A and B are the same target, one can consider whether the target size change data changes abruptly during the movement from target A to target B; simultaneously, the angle between vector A and vector A2, and the angle between vector A2 and vector B can be calculated. If the size change data does not change abruptly and the angle change is less than a preset threshold, then targets A and B are considered the same target. When determining whether targets A and C are the same target, by calculating the angle between vector A and vector A1, and the angle between vector A1 and vector C, it can be determined that after passing an obstruction (e.g., ... Figure 4bWhen encountering tree-shaped obstructions, the direction deviates significantly, which is inconsistent with reality. Therefore, target A and target C are not the same target.

[0071] S403. If the third moving target and the fourth moving target are the same, then the missed video frame is determined based on the frame identifier code of the video frame containing the third moving target and the frame identifier code of the video frame containing the fourth moving target.

[0072] Optionally, to determine the missed video frames, it is necessary to first determine the video frames before and after the missed detection occurred, that is, to determine the last video frame that includes the third moving target (i.e., target A) and the first video frame that includes the fourth moving target (i.e., target B). Then, based on the frame identifier code (i.e., frame number) of the two video frames, the number and frame number of the missed video frames can be determined.

[0073] S404. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, mark the missed detection area in the missed video frame.

[0074] S405. Verify the missed detection region based on the motion region detected from the missed video frame.

[0075] The process of calibrating and verifying the missed detection areas is described in the above embodiments and will not be repeated here.

[0076] In this embodiment of the present disclosure, it is possible to quickly detect whether a missed detection has occurred based on the motion parameters of the moving target, and when a missed detection occurs, the missed video frame can be located quickly and accurately based on the frame identifier code of the video frame.

[0077] Figure 5a This is a flowchart illustrating another method for detecting missed video frames according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See below. Figure 5a The specific process for detecting missed video frames is as follows:

[0078] S501. Based on the moving targets detected from the monitored video frames, determine the missed video frames.

[0079] The process for identifying missed video frames can be found in the above embodiments and will not be repeated here.

[0080] S502. Determine the position weighting coefficient of the missed area in the missed video frame based on the frame identifier codes of the previous and next frames of the missed video frame.

[0081] For example, see Figure 5bThe diagram illustrates a series of consecutive video frames, including those with missed detections. As shown, a missed detection occurs between frame numbers (i.e., frame identifiers) 0000x and 0000y. The video frames between these frames are considered missed detections. Frame 0000A is the frame preceding the missed detection, and frame 0000B is the frame following it. Optionally, the relative frame number σ of the missed detection is used as a position weighting coefficient. The relative frame number is a positive integer greater than zero and is less than the difference between frame numbers 0000A and 0000B. That is, the relative frame number of the missed detection is {σ|σ∈N}. + , σ<BA}. For example, the position weighting coefficient of frame number 0000x is 1, the position weighting coefficient of frame number 0000x+1 is 2, and so on, so that the position weighting coefficient of each missed video frame can be determined.

[0082] S503. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, as well as the position weighting coefficient, mark the missed detection area in the missed video frame.

[0083] For example, in frame number 0000A, the coordinates of the upper left and lower right corners of the moving target are (X... AT Y AT ), (X AB Y AB In frame 0000B, the coordinates of the upper left and lower right corners of the moving target are (X...). BT Y BT ), (X BB Y BB The general coordinate expressions for the top-left and bottom-right corners of the missed detection area in each missed detection video frame are:

[0084]

[0085]

[0086] Therefore, by using a general coordinate expression, the coordinates of the upper left and lower right corners of the missed detection area in each missed detection video frame can be obtained, thus completing the calibration of the missed detection area in each missed detection video frame.

[0087] S504. Verify the missed detection region based on the motion region detected from the missed video frame.

[0088] In this embodiment of the disclosure, based on the position parameters of the moving target in the previous and next frames of the missed video frame, as well as the position weighting coefficient, the missed detection region in the missed video frame can be quickly identified, providing a guarantee for subsequent optimization of the target detection algorithm model.

[0089] Figure 6This is a flowchart illustrating another method for detecting missed video frames according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See below. Figure 6 The specific process for detecting missed video frames is as follows:

[0090] S601. Determine the missed video frames based on the attribute parameters or motion parameters of the moving targets in the monitored video frames.

[0091] S602. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, mark the missed detection area in the missed video frame.

[0092] In this embodiment of the disclosure, if a moving target enters a completely occluded environment and then exits from the completely occluded environment, and if the missed detection area is identified according to steps S601 and S602, there is a possibility that the environmental image area may be mistakenly identified as a missed detection area. Therefore, it is necessary to verify this situation, and the verification process can be found in steps S603-S605.

[0093] S603. Calculate the confidence level of the missed detection area based on the area of ​​the motion area and the area of ​​the missed detection area.

[0094] S604. If the confidence level is greater than or equal to a preset threshold, then the calibration of the missed detection area is determined to be accurate.

[0095] S605. If the confidence level is less than the preset threshold, otherwise, determine that the missed detection area is incorrectly calibrated.

[0096] In this embodiment, motion detection can be triggered when a missed detection is determined to occur, in order to detect the moving region in the missed video frame. Alternatively, motion detection can be performed simultaneously with target detection to determine the moving region in each video frame, thereby obtaining the moving region in the missed video frame. For the missed region calibrated by S602, the area of ​​the missed region can be calculated using its coordinates, and the ratio of the area of ​​the moving region to the area of ​​the missed region is used as the confidence level of the missed region. When the ratio is greater than or equal to a preset threshold, the missed region is determined to be accurately calibrated; otherwise, the calibration is incorrect, and the calibration result is discarded.

[0097] In this embodiment of the disclosure, by verifying the calibration results, erroneous calibration is avoided, thereby preventing the subsequent optimization of the target detection algorithm using images with erroneous calibration data.

[0098] Figure 7 This is a flowchart illustrating another method for detecting missed video frames according to an embodiment of the present disclosure. This embodiment is an optimization based on the above embodiment. See below. Figure 7 The specific process for detecting missed video frames is as follows:

[0099] S701. Determine the missed video frames based on the attribute parameters or motion parameters of the moving targets in the monitored video frames.

[0100] S702. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, mark the missed detection area in the missed video frame.

[0101] S703. Perform image processing on the calibrated missed areas in the missed video frames to obtain an image with calibration data.

[0102] In this embodiment of the disclosure, the missed areas in the missed video frames are processed by image editing. The purpose is to convert the video stream data of the missed areas into image format data. That is, the image editing process can obtain an image with calibration data, wherein the calibration data includes at least the type of moving target.

[0103] S704. Upload the images to the cloud server. When the number of images received by the cloud server reaches a threshold, the cloud server sends the images to the model training server so that the model training server can optimize the object detection algorithm model based on the images and feed the optimized object detection algorithm model back to the cloud server.

[0104] S705: Receives and deploys the optimized target detection algorithm model transmitted from the cloud server.

[0105] In this embodiment, after obtaining the image via S703, the network camera sends the image information to a cloud server for storage. If the network camera has no network connection but the automatic error detection function is enabled, the image is first cached in a front-end storage path (SD card, NAS, etc.). Once the network connection is restored, the image is sent to the cloud server for storage.

[0106] After receiving pre-labeled images from the corresponding network camera, the cloud server collects the image information in the cloud. When the collected image count meets the requirements for optimizing the object detection algorithm model, the image information is sent to the model training server for optimization training. Once training is complete, the trained model is transmitted back to the cloud server, which then notifies the network camera to upgrade its model. After the user confirms the upgrade, the cloud server sends the object detection algorithm model file to the network camera to replace its original model file. After the upgrade, the network camera achieves targeted optimization for missed objects in its environment, thereby improving the algorithm's performance in different environments.

[0107] It should be noted that the training server platform comes pre-loaded with some mainstream object detection algorithm models (YOLO, SSD, FASTRCNN) and network structures (VGG, ResNet, DarkNet) to implement automatic training scripts, which can automatically complete the model training results based on the imported calibration files.

[0108] In this embodiment, images of the identified missed detection areas are generated and uploaded to a cloud server, achieving the goal of timely collection of missed detection materials. Based on the identified images, the target detection algorithm model is optimized and trained, so that the probability of missed detection can be reduced after the network camera upgrades its model.

[0109] Figure 8 This is a schematic diagram of a device for detecting missed video frames according to an embodiment of the present disclosure. This embodiment is applicable to situations where a network camera is used to automatically identify and calibrate missed video frames. The device is configured in a network camera, such as... Figure 8 As shown, the device specifically includes:

[0110] The missed detection frame determination module 801 is used to determine missed video frames based on moving targets detected from the monitoring video frames;

[0111] The calibration module 802 is used to calibrate the missed detection area in the missed detection video frame based on the position parameters of the moving target in the previous and next frames of the missed detection video frame.

[0112] The verification module is used to verify the missed detection region based on the motion region detected from the missed video frame.

[0113] Based on the above embodiments, optionally, the missed frame determination module includes:

[0114] The missed frame determination unit is used to determine missed video frames based on the attribute parameters or motion parameters of moving targets in the monitored video frames.

[0115] Based on the above embodiments, optionally, the missed frame determination unit includes:

[0116] The missed detection sensing subunit is used to determine that a video frame has been missed if the target identification codes of the first moving target and the second moving target are different but the attribute parameters are the same.

[0117] The first missed frame determination subunit is used to determine the missed video frame based on the frame identification code of the video frame including the first moving target and the frame identification code of the video frame including the second moving target.

[0118] Based on the above embodiments, optionally, the missed frame determination unit includes:

[0119] The target detection subunit is used to detect in real time whether a fourth moving target appears in the video frame in response to the detection that a third moving target in the monitored video frame has disappeared;

[0120] The judgment subunit is used to determine whether the third and fourth moving targets are the same after the fourth moving target is detected, based on the motion parameters of the third and fourth moving targets.

[0121] The second missed frame determination subunit is used to determine the missed video frame based on the frame identifier code of the video frame including the third moving target and the frame identifier code of the video frame including the fourth moving target if the third moving target and the fourth moving target are determined to be the same moving target.

[0122] Based on the above embodiments, optionally, the motion parameters include at least one of the following: the size of the moving target in the video frame, the displacement vector of the moving target, and the position of the moving target;

[0123] Accordingly, the specific application of the judgment subunit is as follows:

[0124] Based on the size change data of the third and fourth moving targets, the angle between the displacement vectors of the third and fourth moving targets, or the positional relationship between the third and fourth moving targets, determine whether the third and fourth moving targets are the same.

[0125] Based on the above embodiments, optionally, the calibration module includes:

[0126] The coefficient determination unit is used to determine the position weighting coefficient of the missed area in the missed video frame based on the frame identifier codes of the previous and next frames of the missed video frame.

[0127] The calibration unit is used to calibrate the missed detection area in the missed detection video frame based on the position parameters of the moving target in the previous and next frames of the missed detection video frame, as well as the position weighting coefficient.

[0128] Based on the above embodiments, optionally, the verification module further includes:

[0129] A confidence calculation unit is used to calculate the confidence level of the missed detection region based on the area of ​​the motion region and the area of ​​the missed detection region.

[0130] The first verification unit is used to determine that the missed detection area is accurately calibrated if the confidence level is greater than or equal to a preset threshold.

[0131] The second verification unit is used to determine that the missed area is incorrectly labeled if the confidence level is less than a preset threshold.

[0132] Optionally, based on the above embodiments, the device further includes:

[0133] The image editing module is used to perform image editing on the marked missed areas in the missed video frames to obtain images with calibration data;

[0134] The image upload module is used to upload images to the cloud server. When the number of images received by the cloud server reaches a threshold, the cloud server sends the images to the model training server so that the model training server can optimize the object detection algorithm model based on the images and feed the optimized object detection algorithm model back to the cloud server.

[0135] The receiving and deployment module is used to receive and deploy the optimized detection algorithm model transmitted from the cloud server.

[0136] The apparatus provided in this disclosure can execute the method for detecting missed video frames provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method for detecting missed video frames. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this disclosure.

[0137] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. In this embodiment, the electronic device is exemplarily a network camera device. Figure 9 The structure shown in this embodiment of the electronic device includes: one or more processors 902 and a memory 901; the processors 902 in the electronic device may be one or more. Figure 9 Taking a processor 902 as an example; memory 901 is used to store one or more programs; one or more programs are executed by one or more processors 902, so that one or more processors 902 implement the method for detecting missed video frames as described in any of the embodiments of this disclosure.

[0138] The electronic device may also include an input device 903 and an output device 904.

[0139] The processor 902, memory 901, input device 903, and output device 904 in this electronic device can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0140] The memory 901 in this electronic device serves as a computer-readable storage medium, which can be used to store one or more programs, including software programs, computer-executable programs, and modules. The processor 902 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 901, thereby implementing the method for detecting missed video frames described in the above embodiment.

[0141] Memory 901 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, memory 901 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory 901 may further include memory remotely located relative to processor 902, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0142] Input device 903 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 904 may include display devices such as a display screen.

[0143] Furthermore, when one or more programs included in the aforementioned electronic device are executed by one or more processors 902, the programs perform the following operations:

[0144] Based on the moving targets detected in the surveillance video frames, the missed video frames are identified;

[0145] Based on the position parameters of the moving target in the previous and next frames of the missed video frame, the missed detection area is marked in the missed video frame.

[0146] The missed detection region is verified based on the motion region detected from the missed video frame.

[0147] Of course, those skilled in the art will understand that when one or more programs included in the above-mentioned electronic device are executed by one or more processors, the programs can also perform related operations in the method for detecting missed video frames provided in any embodiment of this disclosure.

[0148] One embodiment of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to perform a method for detecting missed video frames, the method comprising:

[0149] Based on the moving targets detected in the surveillance video frames, the missed video frames are identified;

[0150] Based on the position parameters of the moving target in the previous and next frames of the missed video frame, the missed detection area is marked in the missed video frame.

[0151] The missed detection region is verified based on the motion region detected from the missed video frame.

[0152] The computer storage medium of this disclosure can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable CD-ROM, optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0154] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency (RF), etc., or any suitable combination thereof.

[0155] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (e.g., including local area networks (LANs) or wide area networks (WANs)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] Note that the above description is merely a preferred embodiment and the technical principles employed in this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, it is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this disclosure, and the scope of this disclosure is determined by the scope of the appended claims.

Claims

1. A method for detecting missed video frames, characterized in that, include: Based on the moving targets detected in the surveillance video frames, the missed video frames are identified; Based on the position parameters of the moving target in the previous and next frames of the missed video frame, the missed detection area is marked in the missed video frame; The missed detection region is verified based on the motion region detected from the missed video frame; Among these, the process of identifying missed video frames based on moving targets detected in the monitored video frames includes: Based on the attribute parameters or motion parameters of moving targets in the monitored video frames, identify the missed video frames; The step of calibrating the missed detection region in the missed detection video frame based on the position parameters of the moving target in the preceding and following frames includes: Based on the frame identifier codes of the preceding and following frames of the missed video frame, determine the position weighting coefficient of the missed area in the missed video frame. Based on the position parameters of the moving target in the previous and next frames of the missed video frame, and the position weighting coefficient, the missed detection area is marked in the missed video frame.

2. The method according to claim 1, characterized in that, Based on the attribute parameters of moving targets in the monitored video frames, the missed video frames are identified, including: If the target identification codes of the first moving target and the second moving target are different but the attribute parameters are the same, it is determined that a video frame has been missed. The missed video frames are determined based on the frame identifier codes of the video frames containing the first moving target and the video frames containing the second moving target.

3. The method according to claim 1, characterized in that, Based on the motion parameters of moving targets in the monitored video frames, the missed video frames are identified, including: In response to the detection that a third moving target in a monitored video frame has disappeared, the system continuously detects whether a fourth moving target appears in the video frame. After detecting the fourth moving target, it is determined whether the third moving target and the fourth moving target are the same based on the motion parameters of the third moving target and the fourth moving target; If they are the same, then the missed video frame is determined based on the frame identifier code of the video frame including the third moving target and the frame identifier code of the video frame including the fourth moving target.

4. The method according to claim 3, characterized in that, in, The motion parameters include at least one of the following: the size of the moving target in the video frame, the displacement vector of the moving target, and the position of the moving target; Accordingly, based on the motion parameters of the third moving target and the fourth moving target, determining whether the third moving target and the fourth moving target are the same includes: Based on the size change data of the third moving target and the fourth moving target, the angle between the displacement vectors of the third moving target and the fourth moving target, or the positional relationship between the third moving target and the fourth moving target, determine whether the third moving target and the fourth moving target are the same.

5. The method according to claim 1, characterized in that, The verification of the missed detection region based on the motion region detected from the missed video frame includes: Calculate the confidence level of the missed detection region based on the area of ​​the motion region and the area of ​​the missed detection region; If the confidence level is greater than or equal to a preset threshold, then the missed detection area is determined to be accurately calibrated. If the confidence level is less than a preset threshold, then the missed detection area is determined to be incorrectly labeled.

6. The method according to claim 1, characterized in that, The method further includes: The marked missed detection areas in the missed detection video frames are processed to obtain images with calibration data; The image is uploaded to the cloud server. When the number of images received by the cloud server reaches a threshold, the image is sent to the model training server. The model training server then optimizes the object detection algorithm model based on the image and feeds back the optimized object detection algorithm model to the cloud server. Receive and deploy the optimized target detection algorithm model transmitted by the cloud server.

7. A device for detecting missed video frames, characterized in that, The device includes: The missed detection frame determination module is used to determine missed video frames based on moving targets detected from the monitoring video frames; The calibration module is used to calibrate the missed detection area in the missed detection video frame based on the position parameters of the moving target in the previous and next frames of the missed detection video frame. The verification module is used to verify the missed detection region based on the motion region detected from the missed video frame; The missed frame determination module includes: The missed frame determination unit is used to determine the missed video frames based on the attribute parameters or motion parameters of the moving targets in the monitored video frames. The calibration module includes: The coefficient determination unit is used to determine the position weighting coefficient of the missed area in the missed video frame based on the frame identifier codes of the previous and next frames of the missed video frame. The calibration unit is used to calibrate the missed detection area in the missed detection video frame based on the position parameters of the moving target in the previous and next frames of the missed detection video frame, as well as the position weighting coefficient.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images