Video processing method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a video processing method and device, electronic equipment and storage medium, the method comprising: performing motion target detection on a first video frame sequence to obtain a candidate region of the first video frame sequence, and removing an interference region in the candidate region to obtain a target region of the first video frame sequence; then performing target tracking on the first video frame sequence according to the target region of the first video frame sequence to obtain a target state array after target tracking; and generating an alarm information according to a tracking time of at least one recorded target in the target state array. The embodiments of the present disclosure can improve the accuracy and recall rate of target detection, and further generate more accurate alarm information.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video processing method and apparatus, electronic device and storage medium. Background Technology

[0002] Currently, video surveillance technology is becoming increasingly mature and can be applied to various scenarios, such as daily home monitoring, daily shopping mall monitoring, and monitoring of catering service venues. Furthermore, with the development and progress of video analysis and artificial intelligence technologies, more and more events can be detected and identified in real time through video surveillance. Through video analysis, target objects in the monitored scene can be detected, and the traces of target objects can be recorded, detected in advance, and alarms can be triggered.

[0003] However, in related technologies, for small targets with low contrast to the background, such as mice in the kitchen at night, the monitoring algorithms used by video surveillance have difficulty capturing such targets and effectively tracking and monitoring them. Summary of the Invention

[0004] This disclosure proposes a video processing technology solution.

[0005] According to one aspect of this disclosure, a video processing method is provided, comprising: performing moving target detection on a first video frame sequence to obtain candidate regions of the first video frame sequence; removing interference regions from the candidate regions of the first video frame sequence to obtain target regions of the first video frame sequence; performing target tracking on the first video frame sequence based on the target regions of the first video frame sequence to obtain a target state array after target tracking, the target state array including at least one recorded target, and position information, start time, and end time corresponding to each recorded target; generating alarm information based on the tracking time of at least one recorded target in the target state array, wherein the tracking time of each recorded target is the difference between the end time and the start time corresponding to the recorded target.

[0006] In one possible implementation, removing interfering regions from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence includes: acquiring text prompt information, the text prompt information including at least one negative sample text for indicating the category of the interfering region and a positive sample text for indicating the category of the target region; inputting the text prompt information and the candidate regions of the first video frame sequence into a classification model to obtain the confidence level of each candidate region in the first video frame sequence belonging to the target region; and removing the interfering regions with confidence levels less than a preset threshold from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence.

[0007] In one possible implementation, moving target detection is performed on a first video frame sequence to obtain candidate regions of the first video frame sequence, including: inputting the current frame image in the first video frame sequence into a pre-established Gaussian mixture model for foreground detection to obtain a foreground image; updating the parameters of the Gaussian mixture model based on the current frame image in the first video frame sequence; and denoising the foreground image to generate candidate regions of the current frame image.

[0008] In one possible implementation, target tracking is performed on the first video frame sequence based on the target region of the first video frame sequence to obtain a target state array after target tracking. This includes: performing Kalman filtering on the target region of the current frame image in the first video frame sequence to obtain the position information of at least one predicted target in the current frame image; if any predicted target in the current frame image successfully matches the recorded target feature in the target state array, updating the position information of the corresponding recorded target in the target state array based on the position information of the predicted target, and updating the end time of the corresponding recorded target in the target state array based on the shooting time of the current frame image; or, if any predicted target in the current frame image fails to match the recorded target feature in the target state array, adding a new recorded target to the target state array, and determining the start time of the new recorded target in the target state array based on the shooting time of the current frame image.

[0009] In one possible implementation, the method further includes: deleting the recorded target from the target state array if the predicted target of K consecutive frames fails to match the features of the recorded target.

[0010] In one possible implementation, the method further includes: performing image preprocessing on the video frame sequence to be processed to obtain a first video frame sequence, wherein the image preprocessing includes: performing text detection on the video frame sequence to be processed to determine text regions in the video frame sequence to be processed; removing the text regions in the video frame sequence to be processed to obtain a second video frame sequence; and performing denoising processing on the second video frame sequence to obtain the denoised first video frame sequence.

[0011] In one possible implementation, the target category includes mice, and the interference area category includes at least one of camera text, point light source, water accumulation, and fan, and the interference area includes at least one of camera text area, point light source area, water accumulation area, and fan area.

[0012] According to one aspect of this disclosure, a video processing apparatus is provided, comprising: a detection module for detecting moving targets in a first video frame sequence to obtain candidate regions of the first video frame sequence; a removal module for removing interference regions from the candidate regions of the first video frame sequence to obtain target regions of the first video frame sequence; a tracking module for tracking targets in the first video frame sequence based on the target regions of the first video frame sequence to obtain a target state array after target tracking, the target state array including at least one recorded target and position information, start time, and end time corresponding to each recorded target; and a generation module for generating alarm information based on the tracking time of at least one recorded target in the target state array, wherein the tracking time of each recorded target is the difference between the end time and the start time corresponding to the recorded target.

[0013] In one possible implementation, the removal module is configured to: acquire text prompt information, the text prompt information including at least one negative sample text indicating the category of the interference region and a positive sample text indicating the category of the target region; input the text prompt information and the candidate regions of the first video frame sequence into a classification model to obtain the confidence level of each candidate region in the first video frame sequence belonging to the target region; and remove the interference regions with confidence levels less than a preset threshold from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence.

[0014] In one possible implementation, the detection module is configured to: input the current frame image in the first video frame sequence into a pre-established Gaussian mixture model for foreground detection to obtain a foreground image; and update the parameters of the Gaussian mixture model based on the current frame image in the first video frame sequence; and denoise the foreground image to generate candidate regions for the current frame image.

[0015] In one possible implementation, the tracking module is configured to: perform Kalman filtering on the target region of the current frame image in the first video frame sequence to obtain the position information of at least one predicted target in the current frame image; if any predicted target in the current frame image successfully matches the recorded target feature in the target state array, update the position information of the corresponding recorded target in the target state array according to the position information of the predicted target, and update the end time of the corresponding recorded target in the target state array according to the shooting time of the current frame image; or, if any predicted target in the current frame image fails to match the recorded target feature in the target state array, add a new recorded target to the target state array, and determine the start time of the new recorded target in the target state array according to the shooting time of the current frame image.

[0016] In one possible implementation, the tracking module is further configured to: delete the recorded target from the target state array if the predicted target of K consecutive frames fails to match the features of the recorded target.

[0017] In one possible implementation, the apparatus further includes a preprocessing module, which is configured to: perform image preprocessing on the video frame sequence to be processed to obtain a first video frame sequence, wherein the image preprocessing includes: performing text detection on the video frame sequence to be processed to determine text regions in the video frame sequence to be processed; removing the text regions in the video frame sequence to be processed to obtain a second video frame sequence; and performing denoising processing on the second video frame sequence to obtain the denoised first video frame sequence.

[0018] In one possible implementation, the target category includes mice, and the interference area category includes at least one of camera text, point light source, water accumulation, and fan, and the interference area includes at least one of camera text area, point light source area, water accumulation area, and fan area.

[0019] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0020] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0021] In this embodiment, moving target detection can be performed on a first video frame sequence to obtain candidate regions of the first video frame sequence. Interference regions within the candidate regions are then removed to obtain the target regions of the first video frame sequence. Then, target tracking can be performed on the first video frame sequence based on the target regions, resulting in a target state array after target tracking. Alarm information is generated based on at least one record in the target state array indicating the target's tracking time. This approach first preliminarily determines candidate regions of suspected targets through moving target detection, then performs an initial filtering of these candidate regions to remove interference regions that are not target regions, and finally performs a secondary filtering in the time domain using a target tracking method. This improves target detection accuracy and recall, thereby generating more accurate alarm information.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0024] Figure 1 A flowchart illustrating a video processing method according to an embodiment of the present disclosure is shown.

[0025] Figure 2 A schematic diagram of a classification model according to an embodiment of the present disclosure is shown.

[0026] Figure 3 A schematic diagram of the target area according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown.

[0028] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0029] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0030] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0031] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0032] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0033] Figure 1A flowchart illustrating a video processing method according to an embodiment of the present disclosure is shown, such as... Figure 1 As shown, the video processing method includes:

[0034] In step S11, moving target detection is performed on the first video frame sequence to obtain candidate regions of the first video frame sequence.

[0035] In step S12, interference regions in the candidate regions of the first video frame sequence are removed to obtain the target region of the first video frame sequence.

[0036] In step S13, target tracking is performed on the first video frame sequence according to the target region of the first video frame sequence to obtain a target state array after target tracking. The target state array includes at least one recorded target, as well as the position information, start time, and end time corresponding to each recorded target.

[0037] In step S14, an alarm message is generated based on the tracking time of at least one recorded target in the target state array, wherein the tracking time of each recorded target is the difference between the end time and the start time corresponding to the recorded target.

[0038] In one possible implementation, the video processing method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0039] In one possible implementation, the video processing method can be implemented by a processor calling computer-readable instructions stored in memory. In one example, the processor can be a general-purpose processor such as a central processing unit (CPU), a graphics processing unit (GPU), or an application-specific integrated circuit (ASIC), or it can be an artificial intelligence processor, such as an artificial intelligence (AI) chip, for example, a neural processing unit (NPU).

[0040] In one possible implementation, the video processing method of this disclosure can be applied to: capturing small target objects with low contrast to the background from a video to be processed. For example, detecting mice from a nighttime kitchen surveillance recording, where the mouse has a small area and low contrast to the background in the grayscale image of the nighttime kitchen surveillance recording.

[0041] In one possible implementation, the first video frame sequence is obtained from the video to be processed collected from the location to be detected. The location to be detected can be a catering service establishment, or a company, shopping mall, hospital, school, station, stadium, factory, warehouse, home kitchen, etc. This disclosure does not impose any restrictions on the location to be detected.

[0042] For example, the video to be processed can be any video about the location to be inspected acquired by the electronic device. The electronic device can acquire video of the current location to be inspected using a video sensor deployed at the location to be inspected, obtaining one or more videos to be processed. Alternatively, the electronic device can select one or more videos about the location to be inspected from multiple videos included in a video file as the videos to be processed. In some implementations, the electronic device can acquire one or more videos about the location to be inspected from other devices. This disclosure does not limit this.

[0043] For example, the video to be processed collected from the location to be detected can be a video file packaged in a certain encapsulation format, such as a Flash Video (FLV) video file, an Advanced Streaming Format (ASF) video file, an Audio Video Interleaved (AVI) video file, a Moving Picture Experts Group 4 (MP4) video file, etc. This disclosure embodiment does not limit the format of the video to be processed.

[0044] Based on the encapsulation format of the video to be processed, a decoding method matching the encapsulation format can be used to parse the video to obtain the first video frame sequence.

[0045] In one possible implementation, in step S11, moving target detection can be performed on the first video frame sequence to obtain candidate regions of the first video frame sequence.

[0046] For example, moving target detection can be performed on the first video frame sequence based on the background model method. For instance, a background model of the first video frame sequence can be established first, and then the moving target can be extracted by subtracting the current video frame image from the background model, thus obtaining candidate regions for the moving target. Moving target detection can rely on a stable background model to obtain relatively complete foreground features, and the moving target can be obtained by comparing the current frame with the background model.

[0047] Among these methods, a background model can be obtained using statistical methods. Furthermore, the background model can be updated in real time according to the feature information of the current video frame image to adapt to changes in lighting and the scene itself. Morphological methods and post-processing such as detecting the area of ​​connected components can be used to eliminate the influence of noise and background disturbances, thereby obtaining more accurate candidate regions for moving targets.

[0048] For example, moving target detection can be performed on the first video frame sequence using artificial intelligence. For instance, the first video frame sequence can be input into a trained deep learning model, which can then automatically perform moving target detection on the first video frame sequence to obtain candidate regions where moving targets may exist. The deep learning model can be a neural network model trained (learned) on a large number of samples, capable of performing moving target detection based on the first video frame sequence.

[0049] For example, the inter-frame difference method can also be used to detect moving targets in the first video frame sequence. By performing difference operations on consecutive video frame images in the first video frame sequence, candidate regions in the first video frame sequence that may contain moving targets can be obtained. Specifically, in consecutive video frame images, if the difference in pixel values ​​of corresponding regions is very small, the region can be considered stationary; otherwise, the region is considered as a candidate region where a moving target is located.

[0050] For example, optical flow can also be used to detect moving targets in the first video frame sequence. For instance, based on the temporal changes of each pixel in the first video frame sequence, a velocity vector can be assigned to each pixel in each video frame, forming an optical flow field. By detecting changes in the optical flow field of the first video frame sequence, candidate regions in the first video frame sequence that may contain moving targets can be obtained.

[0051] It should be understood that the embodiments of this disclosure do not limit the methods for detecting moving targets.

[0052] In step S11, candidate regions of the first video frame sequence are obtained. In step S12, interference regions in the candidate regions of the first video frame sequence are removed to obtain the target region of the first video frame sequence.

[0053] Since the candidate regions of the first video frame sequence are suspected target regions, they typically include not only the target region but also interfering regions. For example, in a scenario of monitoring mice in a kitchen at night, interfering objects such as rising steam and flashing lights can cause false alarms. To remove false alarms, a neural network can be pre-trained to filter out interfering regions, thus removing them from the candidate regions of the first video frame sequence and obtaining the target region of the first video frame sequence.

[0054] The neural network may include at least one of the following: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Neural Networks (DNN), YOLO (You Only Look Once) based on deep learning, Residual Networks (ResNets), Back Propagation (BP) neural networks, and Backbone Neural Networks. The network structure described above is merely an example; the neural network may also include multiple convolutional layers, multiple pooling layers, multiple fully connected layers, etc. The specific construction method and structure of the network can be determined according to actual circumstances. The above example does not constitute a limitation on the embodiments of this disclosure.

[0055] After obtaining the target region of the first video frame sequence in step S12, target tracking can be performed on the first video frame sequence in step S13 based on the target region of the first video frame sequence to obtain the target state array after target tracking.

[0056] For example, for any current frame image in the first video frame sequence, target features can be extracted from one or more target regions present in the current frame image to obtain the target features corresponding to each target region in the current frame image. Then, in subsequent frame images, target regions that have the same or most similar features to each target region in the current frame image are found.

[0057] In this way, in the first video frame sequence, feature matching is performed on each target region in each video frame image in the direction from the first frame to the last frame, so that multi-target tracking of the first video frame sequence can be realized. Multiple target regions belonging to the same target object correspond to a tracked target, which can be recorded in the target state array. The position information, start time, and end time of each recorded target can be recorded synchronously in the target state array.

[0058] The location information is used to represent the trajectory of the recorded target in the first video frame sequence, such as a set of location information of multiple target areas corresponding to the recorded target; the start time indicates the time when the recorded target first appears in the first video frame sequence; and the end time indicates the time when the recorded target last appears in the first video frame sequence.

[0059] After obtaining the target status array in step S13, an alarm message can be generated in step S14 based on the tracking time of at least one recorded target in the target status array.

[0060] For example, assume that there are records of target 1 to target N (N≥1) in the target state array, where target 1 corresponds to location information 1, start time 1 and end time 1, target 2 corresponds to location information 2, start time 2 and end time 2, and so on, and target N corresponds to location information N, start time N and end time N.

[0061] For any recording target i (i∈[1,N]) from recording target 1 to recording target N, the tracking time of recording target i can be obtained from the difference between the end time i and the start time i corresponding to the recording target i. Alarm information can be generated based on the tracking time of recording target i.

[0062] For example, multiple preset intervals can be set, each corresponding to a different alarm level. By analyzing the matching relationship between the tracking time of target i and the preset intervals (e.g., if the tracking time of target i falls within a certain preset interval, it means the tracking time of target i matches that preset interval), alarm information corresponding to different alarm levels can be generated. Higher alarm levels indicate a greater probability that target i is a monitored target (e.g., a mouse).

[0063] For example, if the tracking time of target i is greater than a preset threshold, an alarm message can be generated to indicate that a monitored target (such as a mouse) has been detected; conversely, if the tracking time of target i is less than or equal to the preset threshold, no alarm will be generated.

[0064] It should be understood that in practical applications, alarm rules can be set according to specific application scenarios so as to generate alarm information based on the alarm rules and the tracking time of each recorded target in the target status array. The embodiments disclosed herein do not limit this.

[0065] Electronic devices can send generated alarm information to user terminals or display alarm information through their own components to notify or remind users that a target object has been monitored. The alarm information can be in the form of text, voice, or image, etc., and this disclosure does not limit the alarm method. The alarm information may include the location information, start time, and end time of the corresponding recorded target, or may include alarm sounds or warning voice, text, images, or videos indicating that a target has been detected, etc., and this disclosure does not limit the content of the alarm information.

[0066] In the embodiments of this disclosure, moving target detection can be performed on a first video frame sequence to obtain candidate regions of the first video frame sequence. Interference regions within the candidate regions are then removed to obtain the target regions of the first video frame sequence. Then, target tracking can be performed on the first video frame sequence based on the target regions, resulting in a target state array after target tracking. Alarm information is generated based on at least one record in the target state array indicating the target's tracking time. This approach first preliminarily determines candidate regions of suspected targets through moving target detection, then performs an initial filtering of the candidate regions to remove interference regions that are not target regions, and finally performs a secondary filtering in the time domain using a target tracking method. This improves target detection accuracy and recall, thereby generating more accurate alarm information.

[0067] The video processing method of this disclosure embodiment will be described in detail below.

[0068] In one possible implementation, the video to be processed is parsed to obtain multiple video frames of the video to be processed; according to a preset time interval, frames are extracted from the multiple video frames of the video to be processed to obtain the sequence of video frames to be processed.

[0069] For example, the video to be processed can be a video with a high frame rate. The video to be processed can be parsed and decomposed into multiple video frames according to its frame rate. Then, frame extraction processing can be performed on the above-mentioned multiple video frames. In the example, frames can be randomly extracted or extracted according to a preset rule. For example, frames can be extracted according to a preset time interval, that is, a video frame can be selected every preset time interval according to the timestamp of each video frame (for example, a video frame can be selected every 0.1 seconds). Another example is that frames can be extracted every predetermined number of video frames, for example, a video frame can be selected every two frames (that is, a video frame can be selected every three frames). After the above frame extraction processing, the extracted video frames can form a sequence of video frames to be processed. This disclosure does not limit the specific method of frame extraction.

[0070] In this way, when processing the video, it is not necessary to process every single video frame. Instead, a portion of the video frames can be selected for processing, thereby improving processing efficiency.

[0071] In one possible implementation, before step S11, the method further includes: performing image preprocessing on the video frame sequence to be processed to obtain a first video frame sequence, wherein the image preprocessing includes: performing text detection on the video frame sequence to be processed to determine the text regions in the video frame sequence to be processed; removing the text regions in the video frame sequence to be processed to obtain a second video frame sequence; and performing denoising processing on the second video frame sequence to obtain the denoised first video frame sequence.

[0072] For example, text detection can be performed on the video to be processed to locate the text region in the video frame sequence to be processed, and the pixel values ​​in the detected text region can be assigned to preset values ​​(e.g., the preset value can be 0, the average value of the background of the video to be processed, etc.) to remove the interference of text (such as time watermark) and obtain the second video frame sequence.

[0073] The text detection method can be a regression-based text detection method or a segmentation-based text detection method, and this disclosure does not impose any restrictions on it.

[0074] Then, smoothing filtering is applied to the second video frame sequence. This removes sharp noise from the second video frame sequence without blurring the contours or lines of each video frame, resulting in the processed first video frame sequence. Smoothing filtering uses a template (or convolution, mask) to scan each pixel in each video frame image, replacing the value of the center pixel of the template with the weighted average gray value (or median, minimum, or maximum value) of the pixels in the neighborhood determined by the template. Smoothing filtering methods may include Gaussian filtering, median filtering, local averaging filtering, and K-nearest neighbor averaging filtering. This disclosure does not limit the specific method of smoothing filtering.

[0075] By performing image preprocessing on the video frame sequence to be processed, irrelevant information in the video frame sequence can be eliminated, and useful real information in the video frame sequence can be restored, making the preprocessed first video frame sequence more detectable, which is beneficial to improving the accuracy and efficiency of subsequent detection.

[0076] After obtaining the first video frame sequence after image preprocessing, in step S11, moving target detection can be performed on the first video frame sequence to obtain candidate regions of the first video frame sequence.

[0077] In relevant object detection techniques, deep learning-based two-stage or single-stage object detection methods cannot effectively detect very small objects (such as mice in a kitchen at night) in video frames; single Gaussian background modeling methods are not robust enough and are very sensitive to noise; optical flow-based moving object detection methods have high computational complexity, long computation time, and poor practicality. Inter-frame difference-based moving object detection methods cannot completely extract all relevant feature pixels and are prone to producing holes in moving object regions.

[0078] Therefore, in order to improve the recall rate and detection accuracy, embodiments of this disclosure may employ a Gaussian mixture background modeling method to perform moving target detection on the first video frame sequence and obtain candidate regions of the first video frame sequence.

[0079] In one possible implementation, step S11 may include: inputting the current frame image in the first video frame sequence into a pre-established Gaussian mixture model for foreground detection to obtain a foreground image; and updating the parameters of the Gaussian mixture model based on the current frame image in the first video frame sequence; and denoising the foreground image to generate candidate regions for the current frame image.

[0080] For example, a scene image may be obtained for building a Gaussian mixture model, which may include any image of the place to be detected, and this disclosure is not limited thereto.

[0081] Multiple Gaussian models can be used to characterize the features of each pixel in a scene image, constructing and initializing a Gaussian mixture model M to simulate a static scene (background) of the location to be detected. During the construction of the Gaussian mixture model M, multiple Gaussian distributions can be used to construct the model for each pixel in the scene image. For example, in some embodiments, multiple Gaussian distributions for each pixel can be calculated and constructed sequentially, but this is not a limitation.

[0082] To improve detection efficiency, before acquiring the first video frame sequence, a Gaussian mixture model M can be pre-established using the scene image of the location to be detected, and the pre-established Gaussian mixture model M can be directly used to detect moving targets in the first video frame sequence. Alternatively, to improve detection accuracy, the first video frame image in the first video frame sequence can be used as the scene image. Based on the first video frame image in the first video frame sequence, a Gaussian mixture model M can be constructed and initialized, and then the Gaussian mixture model M can be used to detect moving targets in the first video frame sequence.

[0083] Read the current frame image I from the first video frame sequence in sequence. k The current frame image I in the first video frame sequence can be... kThe image is input into a pre-established Gaussian mixture model M for foreground detection, resulting in a foreground image f. k =M(I k ), where the foreground image f k It can be a binary image.

[0084] The Gaussian mixture model M can be adaptively updated over time. For example, it can be based on the current frame image I in the first video frame sequence. k , will the current frame image I k Each pixel value is matched with the Gaussian distribution of the Gaussian mixture model M. If a match is found, the pixel is a background pixel; otherwise, the pixel is detected as a foreground pixel. The parameters of the Gaussian mixture model M are updated based on the matched pixels.

[0085] In this process, the Gaussian mixture model M can be updated first based on the current frame image in the first video frame sequence. Then, the updated Gaussian mixture model M is used to perform foreground detection on the current frame image to obtain the foreground image f. k Alternatively, the Gaussian mixture model M before the update can be used to perform foreground detection on the current frame image to obtain the foreground image f. k Then, the Gaussian mixture model M is updated based on the current frame image, but this disclosure does not limit this process.

[0086] Using a Gaussian mixture model M to perform foreground detection on the current frame image, we obtain the foreground image f. k It can be used for the foreground image f k Denoising is performed to generate candidate regions for the current frame image.

[0087] Optionally, opening and / or closing operations from mathematical morphology can be used to apply the foreground image f. k Denoising is performed to generate candidate regions for the current frame image.

[0088] For example, we can first select the struct element SE for opening and / or closing operations in mathematical morphology, that is:

[0089]

[0090] According to the structuring element SE shown in formula (1), the foreground image f k Perform mathematical morphological opening operations on the foreground image f extracted from the Gaussian mixture model M. k Denoising is performed to remove isolated small dots, burrs, and small bridges, resulting in a denoised foreground image. Still using the above structuring elements, the denoised foreground image is processed. Perform mathematical morphological closing operations to fill in small holes and close small cracks, thus obtaining a new foreground image. As a candidate region for the current frame image.

[0091] For example, the foreground image f can also be determined based on the structuring element SE shown in formula (1). k Perform mathematical morphological closing operations to fill in small holes and close small cracks, obtaining a denoised foreground image. Still using the above structuring elements, the denoised foreground image is processed. Perform mathematical morphological opening operations to remove isolated dots, burrs, and small bridges, resulting in a new foreground image. As a candidate region for the current frame image.

[0092] It should be understood that the embodiments of this disclosure use only the structural element of formula (1) as an example. Different structural elements can be designed according to different application scenarios. This disclosure does not limit the size and value of the structural element.

[0093] Alternatively, transform domain filtering can be used to filter the foreground image f. k Denoising is performed to generate candidate regions for the current frame image. For example, the foreground image f can be used as a candidate region. k Transforming from the spatial domain to the transform domain yields the transformed image s. k Then, the transform coefficients in the transform domain are adjusted, and based on the adjusted transform coefficients, the transformed image s is processed. k Perform an inverse transform to transform the image s k Transforming from the transform domain to the spatial domain to achieve a better understanding of the foreground image f k The goal is to remove noise and obtain candidate regions for the current frame image. Transform domain filtering methods can include Fourier transform filtering, wavelet transform filtering, etc.

[0094] Alternatively, variational methods (such as active contouring) can be used to refine the foreground image f. k Denoising is performed to generate candidate regions for the current frame image. For example, the foreground image f can be determined. k The energy function is minimized to make the foreground image f k Achieve a smooth state and use it as a candidate region for the current frame image.

[0095] Alternatively, a neural network method can be used to process the foreground image f. k Denoising is performed to generate candidate regions for the current frame image. For example, a neural network model for denoising can be pre-trained to convert the foreground image f... k Input the trained neural network model to obtain candidate regions for the current frame image.

[0096] It should be understood that, depending on the actual application scenario and computing resources, an appropriate denoising method can be selected for the foreground image f. k This disclosure does not limit the denoising process.

[0097] In this way, candidate regions corresponding to each current video frame can be obtained sequentially according to the order of the first video frame sequence. The Gaussian mixture model M is then updated in real time based on the image feature information of each current video frame to adapt to changes in lighting and the scene itself, thus improving the recall rate of the detection results. Furthermore, the candidate regions undergo denoising processing, which eliminates the influence of noise and background disturbances, further enhancing the accuracy of the candidate regions.

[0098] In particular, for the application scenario of detecting rats in catering service venues at night, taking advantage of the fact that the target rats can move, the Gaussian mixture background modeling method, which is more robust than single Gaussian background modeling, is used to more easily detect moving rat targets on gray images.

[0099] Since the candidate region of the first video frame sequence obtained in step S11 is the coarse target detection result of the first stage, although it has a good recall rate, it usually includes not only the target region (e.g., the area where the mouse is located), but also interference regions that are not target regions (e.g., water vapor regions, flashing light regions, etc.), and cannot perform semantic discrimination on the detected target. In order to further improve the detection accuracy, in step S12, the interference regions in the candidate region of the first video frame sequence can be removed to obtain the target region of the first video frame sequence.

[0100] In one possible implementation, step S12 may include: acquiring text prompt information, the text prompt information including at least one negative sample text for indicating the category of the interference region and a positive sample text for indicating the category of the target region; inputting the text prompt information and the candidate regions of the first video frame sequence into a classification model to obtain the confidence level of each candidate region in the first video frame sequence belonging to the target region; removing the interference regions with confidence levels less than a preset threshold from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence.

[0101] For example, a user can input different text prompts for different application scenarios through the interactive interface on the electronic device, so that the electronic device can adapt to different application scenarios and obtain one or more text prompts for indicating the category of interference area; or, the user can also select one or more text prompts from the set of text prompts stored in the database of the electronic device, so that the electronic device can adapt to different application scenarios and obtain one or more text prompts for indicating the category of interference area; or, one or more texts that are pre-set by the electronic device to adapt to different application scenarios can be used as text prompts for indicating the category of interference area; this disclosure does not limit the method of obtaining text prompts.

[0102] For example, in an application scenario involving mouse detection at night in a food service establishment, the negative sample text in the prompt message indicates that the interference region category includes at least one of camera text, point light source, puddles, and fan, and the interference region includes at least one of the camera text area, point light source area, puddles area, and fan area. The positive sample text indicates that the target category includes mice, and the target area includes mouse areas.

[0103] Then, the acquired text prompts and candidate regions from the first video frame sequence can be input into a pre-trained classification model to obtain the confidence score of each candidate region belonging to the target region in the first video frame sequence. To improve the detection efficiency of the classification model, the candidate regions of the first video frame sequence can be regularized by extracting the minimum bounding rectangle of each candidate region as the region of interest. The region of interest is then scaled to a preset size w×w before being input into the pre-trained classification model along with the text prompts for classification, yielding a classification confidence score for whether the region is the target object.

[0104] Figure 2 A schematic diagram of a classification model according to an embodiment of the present disclosure is shown, such as Figure 2 As shown, the classification model may include a text encoding module for converting text information into text feature vectors and an image encoding module for converting image information into image feature vectors. By jointly training text and image information, the classification model enables it to learn a mapping relationship during the inference phase, even without training samples of the current category. This allows it to map the sample to the existing vector space and infer the category a previously unseen sample might belong to using distance judgment within the vector space. Specifically, the classification model can transform image classification into image-text matching, mapping image and semantic information to the same multimodal semantic space, and then training using contrastive learning methods.

[0105] During the training phase, the training data for the classification model can include multiple pairs of images and text. For example... Figure 2 As shown in the training phase, assuming each batch of training data contains N (image, text) pairs, N (image, text) pairs will be input into the classification model, resulting in N text feature vectors, i.e., T1~T N And N image feature vectors, namely: I1~I N For N text feature vectors T1 to T2... N Any text feature vector T in i Let i∈[1,N], and let the text feature vector T i With image feature vector I iThe semantic similarity between text feature vectors and image feature vectors should be maximized, while the similarity with other image feature vectors should be minimized. To calculate the semantic similarity between each text feature vector and each image feature vector, it can be viewed as a matrix, where the horizontal axis represents text feature vectors and the vertical axis represents image feature vectors. The labels of the text and image feature vectors are arranged in order, resulting in an N*N square matrix. The position of each element (i,j) in the matrix is ​​T. i with I j The semantic similarity between the elements, i, j∈[1,N], can be obtained by using cosine similarity or dot product. Through multiple iterations of training, the values ​​of the elements on the diagonal are maximized while the values ​​of the remaining elements are minimized, until the classification model converges, resulting in a well-trained classification model.

[0106] During the reasoning phase, a guidance template can be created based on the input text prompts, and the set of text prompts can be mapped to this guidance template. For example... Figure 2 As shown in the inference phase, the guiding template is "This is an image related to {category}". After mapping the set of text prompts to the guiding template, we can obtain the following set: {"This is an image about a mouse", "This is an image about camera text", "This is an image about a point light source", "This is an image about puddles", "This is an image about a fan", etc.}. For the text in the set, after passing through the text encoding module, we can obtain the text feature vector corresponding to each category. Then, for each candidate region to be classified, we compare the semantic similarity between the image feature vector of the candidate region and the text feature vectors corresponding to multiple categories, and select the text information corresponding to the maximum semantic similarity as the classification result of the candidate region. The maximum semantic similarity is used as the classification confidence of the target object to which the candidate region belongs.

[0107] Since a higher confidence score for a candidate region indicates a greater likelihood that it is the target region, and a lower confidence score indicates a lower likelihood, interference regions with excessively low scores can be filtered out based on the confidence score of candidate regions at each potential target location using a preset threshold. For example, for any candidate region in the first video frame sequence, if the confidence score of the candidate region is greater than the preset threshold, it can be used as the target region of the first video frame sequence; if the confidence score of the candidate region is less than or equal to the preset threshold, it indicates that the candidate region is an interference region where the target does not exist, and the candidate region can be deleted.

[0108] Figure 3 A schematic diagram of the target area according to an embodiment of the present disclosure is shown, such as Figure 3As shown, the left part is a current frame image of the first video frame sequence, and the rectangular area on the right is the target area detected from the video frame image.

[0109] In this way, based on a trained classification model, the initial coarse detection results (i.e., candidate regions) can be filtered to remove interfering regions and obtain more accurate target regions. Furthermore, compared to related techniques such as Convolutional Neural Networks (CNNs), template matching, and Support Vector Machines, the classification model-based filtering method offers higher accuracy and better scalability. It only requires changing the text prompts to expand the scope of the target to be detected and the targets to be filtered, allowing for free definition of the positive sample categories to be detected and the negative sample categories to be filtered.

[0110] After obtaining the target region of the first video frame sequence in step S12, target tracking can be performed on the first video frame sequence in step S13 based on the target region of the first video frame sequence to obtain the target state array after target tracking.

[0111] In one possible implementation, step S13 may include:

[0112] Kalman filtering is performed on the target region of the current frame image in the first video frame sequence to obtain the location information of at least one predicted target in the current frame image;

[0113] If any predicted target in the current frame image successfully matches the feature of a recorded target in the target state array, the position information of the corresponding recorded target in the target state array is updated according to the position information of the predicted target, and the end time of the corresponding recorded target in the target state array is updated according to the capture time of the current frame image.

[0114] Alternatively, if any predicted target in the current frame image fails to match the features of a recorded target in the target state array, a new recorded target is added to the target state array, and the start time of the new recorded target in the target state array is determined based on the capture time of the current frame image.

[0115] For example, by performing multi-target tracking on the target region of the current frame image in the first video frame sequence using Kalman filtering, the predicted target position information corresponding to each target region can be obtained. Specifically, Kalman filtering can utilize the target's dynamic information to construct a linear system state equation, thereby removing the influence of noise and obtaining an estimate of the target's position.

[0116] A target state array can be established based on the position information of each predicted target in the first frame image of the first video frame sequence. This target state array can record the position information, start time, end time, etc. of all predicted targets. The start time of each predicted target can be set to the shooting time of the first frame, and the end time can be initially set to empty, waiting to be determined based on subsequent frame images.

[0117] When the predicted target is detected in the next frame, feature matching is performed with the recorded targets already in the target state array. If a match is successful, the position information and end time of the corresponding recorded target in the array are updated. For example, the position information of the predicted target can be updated to the position information of the corresponding recorded target to make the trajectory of the recorded target more accurate, and the capture time of the frame can be replaced with the end time of the recorded target. If a match fails, a new recorded target is added to the target state array, and the position information of the predicted target is updated to the position information of the new recorded target. The capture time of the frame is determined as the start time of this new recorded target.

[0118] Feature matching can be performed based on one or more features of the predicted target and the recorded target, such as color features, texture features, shape features, spatial relationship features, contour features, region features, etc. This disclosure does not impose specific limitations on the feature matching method.

[0119] In this way, the target state array after target tracking can be obtained efficiently and accurately, which is beneficial for subsequent secondary filtering in the time domain based on the target tracking method, thereby further improving the target detection accuracy.

[0120] In one possible implementation, the method further includes: deleting the recorded target from the target state array if the predicted target of K consecutive frames fails to match the features of the recorded target.

[0121] For example, assuming that target A exists in the target state array, if no predicted target is associated with target A in the next K consecutive frames (K is a positive integer), that is, if no predicted target that satisfies feature matching with target A exists in the next K consecutive frames, then target A is deleted from the target state array. The specific value of K can be set according to the actual application scenario, and the embodiments of this disclosure do not impose any limitations on this.

[0122] In this way, interfering targets in the target state array can be further removed, reducing the consumption of computing resources and improving the accuracy of the target state array.

[0123] In step S13, a target state array is obtained. In step S14, an alarm message is generated based on the tracking time of at least one recorded target in the target state array. Where the first video frame sequence is obtained from real-time video of the location to be detected, the target state array can be updated in real-time based on the real-time first video frame sequence, and an alarm can be generated in real-time based on the updated target state array.

[0124] In one possible implementation, in step S14, hierarchical alarms can be generated based on the target state array. If the tracking time of a recorded target in the target state array is less than or equal to a preset threshold, a primary alarm is generated, indicating that the currently recorded target is unlikely to be a monitored target (e.g., a mouse). If the tracking time of a recorded target in the target state array is greater than the preset threshold, a higher-level alarm is generated, indicating that the currently recorded target is likely to be a monitored target (e.g., a mouse).

[0125] In summary, the embodiments of this disclosure enable motion target detection on a first video frame sequence to obtain candidate regions of the first video frame sequence. Interference regions within the candidate regions are then removed to obtain the target regions of the first video frame sequence. Subsequently, target tracking is performed on the first video frame sequence based on the target regions, resulting in a target state array after target tracking. Alarm information is generated based on at least one record in the target state array indicating the target's tracking time. This approach first preliminarily determines candidate regions of suspected targets through motion target detection, then performs initial filtering on these candidate regions to remove interference regions that are not target regions. Finally, a secondary filtering is performed in the time domain using target tracking methods. This improves target detection accuracy and recall, thereby generating more accurate alarm information.

[0126] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0127] In addition, this disclosure also provides video processing apparatus, electronic equipment, computer-readable storage medium, and program, all of which can be used to implement any of the video processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0128] Figure 4 A block diagram of a video processing apparatus according to an embodiment of the present disclosure is shown, such as Figure 4 As shown, the device includes:

[0129] Detection module 41 is used to perform moving target detection on the first video frame sequence to obtain candidate regions of the first video frame sequence;

[0130] The removal module 42 is used to remove interference regions from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence.

[0131] The tracking module 43 is used to track the target in the first video frame sequence according to the target region of the first video frame sequence, and obtain a target state array after target tracking. The target state array includes at least one recorded target, as well as the position information, start time and end time corresponding to each recorded target.

[0132] The generation module 44 is used to generate alarm information based on the tracking time of at least one recorded target in the target state array.

[0133] In one possible implementation, the removal module 42 is configured to: acquire text prompt information, the text prompt information including at least one negative sample text for indicating the category of the interference region and a positive sample text for indicating the category of the target region; input the text prompt information and the candidate regions of the first video frame sequence into a classification model to obtain the confidence level of each candidate region in the first video frame sequence belonging to the target region; and remove the interference regions whose confidence level is less than a preset threshold from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence.

[0134] In one possible implementation, the detection module 41 is configured to: input the current frame image in the first video frame sequence into a pre-established Gaussian mixture model for foreground detection to obtain a foreground image; and update the parameters of the Gaussian mixture model according to the current frame image in the first video frame sequence; and denoise the foreground image to generate candidate regions for the current frame image.

[0135] In one possible implementation, the tracking module 43 is configured to: perform Kalman filtering on the target region of the current frame image in the first video frame sequence to obtain the position information of at least one predicted target in the current frame image; if any predicted target in the current frame image successfully matches the recorded target feature in the target state array, update the position information of the corresponding recorded target in the target state array according to the position information of the predicted target, and update the end time of the corresponding recorded target in the target state array according to the shooting time of the current frame image; or, if any predicted target in the current frame image fails to match the recorded target feature in the target state array, add a new recorded target to the target state array, and determine the start time of the new recorded target in the target state array according to the shooting time of the current frame image.

[0136] In one possible implementation, the tracking module 43 is further configured to: delete the recorded target from the target state array if the predicted target of K consecutive frames fails to match the features of the recorded target.

[0137] In one possible implementation, the apparatus further includes a preprocessing module, which is configured to: perform image preprocessing on the video frame sequence to be processed to obtain a first video frame sequence, wherein the image preprocessing includes: performing text detection on the video frame sequence to be processed to determine text regions in the video frame sequence to be processed; removing the text regions in the video frame sequence to be processed to obtain a second video frame sequence; and performing denoising processing on the second video frame sequence to obtain the denoised first video frame sequence.

[0138] In one possible implementation, the target category includes mice, and the interference area category includes at least one of camera text, point light source, water accumulation, and fan, and the interference area includes at least one of camera text area, point light source area, water accumulation area, and fan area.

[0139] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0140] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0141] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0142] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0143] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0144] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0145] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0146] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0147] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0148] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0149] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0150] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0151] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0152] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0153] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0155] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0156] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0157] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0158] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0159] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video processing method, characterized in that, include: Moving target detection is performed on the first video frame sequence to obtain candidate regions of the first video frame sequence; Remove the interfering regions from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence; Based on the target region of the first video frame sequence, target tracking is performed on the first video frame sequence to obtain a target state array after target tracking. The target state array includes at least one recorded target, as well as the position information, start time, and end time corresponding to each recorded target. An alarm message is generated based on the tracking time of at least one recorded target in the target state array, wherein the tracking time of each recorded target is the difference between the end time and the start time corresponding to the recorded target; The process of removing interference regions from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence includes: Obtain text prompt information, the text prompt information including at least one negative sample text for indicating the category of interference area, and a positive sample text for indicating the category of target; the target category includes a mouse, the interference area category includes at least one of camera text, point light source, water accumulation, and fan, and the interference area includes at least one of camera text area, point light source area, water accumulation area, and fan area; The text prompt information and the candidate regions of the first video frame sequence are input into the classification model to obtain the confidence level of each candidate region in the first video frame sequence belonging to the target region; In the candidate region of the first video frame sequence, the interference regions with confidence levels less than a preset threshold are removed to obtain the target region of the first video frame sequence.

2. The method according to claim 1, characterized in that, Moving target detection is performed on the first video frame sequence to obtain candidate regions of the first video frame sequence, including: The current frame image in the first video frame sequence is input into a pre-established Gaussian mixture model for foreground detection to obtain the foreground image, and... Update the parameters of the Gaussian mixture model based on the current frame image in the first video frame sequence; The foreground image is denoised to generate candidate regions for the current frame image.

3. The method according to claim 1, characterized in that, Based on the target region of the first video frame sequence, target tracking is performed on the first video frame sequence to obtain a target state array after target tracking, including: Kalman filtering is performed on the target region of the current frame image in the first video frame sequence to obtain the location information of at least one predicted target in the current frame image; If any predicted target in the current frame image successfully matches the feature of a recorded target in the target state array, the position information of the corresponding recorded target in the target state array is updated according to the position information of the predicted target, and the end time of the corresponding recorded target in the target state array is updated according to the capture time of the current frame image. or, If any predicted target in the current frame image fails to match the features of a recorded target in the target state array, a new recorded target is added to the target state array, and the start time of the new recorded target in the target state array is determined based on the capture time of the current frame image.

4. The method according to claim 3, characterized in that, The method further includes: deleting the recorded target from the target state array if the predicted target of K consecutive frames fails to match the features of the recorded target.

5. The method according to claim 1, characterized in that, The method further includes: performing image preprocessing on the video frame sequence to be processed to obtain a first video frame sequence, wherein the image preprocessing includes: Perform text detection on the video frame sequence to be processed to determine the text regions in the video frame sequence to be processed; Remove the text region from the video frame sequence to be processed to obtain a second video frame sequence; The second video frame sequence is denoised to obtain the first video frame sequence after denoising.

6. A video processing apparatus, characterized in that, include: The detection module is used to perform moving target detection on the first video frame sequence to obtain candidate regions of the first video frame sequence; The removal module is used to remove interference regions from the candidate regions of the first video frame sequence to obtain the target region of the first video frame sequence. The tracking module is used to track the target in the first video frame sequence according to the target region of the first video frame sequence, and obtain a target state array after target tracking. The target state array includes at least one recorded target, as well as the position information, start time and end time corresponding to each recorded target. The generation module generates alarm information based on the tracking time of at least one recorded target in the target status array, wherein the tracking time of each recorded target is the difference between the end time and the start time corresponding to the recorded target; The removal module is used for: Obtain text prompt information, the text prompt information including at least one negative sample text for indicating the category of interference area, and a positive sample text for indicating the category of target; the target category includes a mouse, the interference area category includes at least one of camera text, point light source, water accumulation, and fan, and the interference area includes at least one of camera text area, point light source area, water accumulation area, and fan area; The text prompt information and the candidate regions of the first video frame sequence are input into the classification model to obtain the confidence level of each candidate region in the first video frame sequence belonging to the target region; In the candidate region of the first video frame sequence, the interference regions with confidence levels less than a preset threshold are removed to obtain the target region of the first video frame sequence.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images