Monitoring image processing system and method based on big data

Abstract

The invention discloses a monitoring image processing system and method based on big data, and relates to the technical field of image processing systems.After N large-size seamless panoramic images are generated through an image splicing module, a background model building module builds a background model of a monitoring area through the N panoramic images based on a Gaussian mixture algorithm; when a moving target appears in the background model, the target detection module extracts the interested moving target from the background image through a moving target detection algorithm, and the target tracking module tracks the moving target in the background model through a particle filter tracking algorithm and records a target walking track; and the abnormity warning module performs abnormity analysis on the behavior of the moving target in the binarized foreground image, and when the abnormal behavior of the moving target is analyzed, a warning is sent to the moving target, and warning information is sent to a manager. The processing system can actively analyze the monitoring function, automatically early warns, and improves the monitoring effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing systems, and in particular to a monitoring image processing system and method based on big data. Background Art

[0002] The rise of big data technologies enables organizations to process and store large amounts of data, including structured data (such as database records) and unstructured data (such as text, images, and videos). Surveillance image processing systems can benefit from big data technologies because they usually need to process large amounts of image and video streams to provide useful information and insights.

[0003] With the continuous development of camera technology and the reduction of costs, cameras have been widely used in various scenarios, such as urban monitoring, commercial monitoring, traffic cameras, etc., which has led to the generation of a large amount of monitoring images and video data, which needs to be effectively processed and analyzed;

[0004] An image processing system is a system that integrates big data technology and image processing technology. It aims to achieve efficient management, analysis and application of large amounts of image and video data captured by surveillance cameras. This system is commonly used in various fields such as security monitoring, traffic management, industrial production, healthcare, etc.

[0005] The prior art has the following deficiencies:

[0006] At present, many monitoring systems mainly rely on connecting cameras to TV monitors and video recorders for remote monitoring and recording. Although this method helps to improve the working environment and provide relevant video records, it lacks real-time, uninterrupted monitoring and analysis of the monitoring scene. Therefore, even if the system can capture potential abnormal situations, it cannot issue an alarm in time, making it difficult to ensure safety. The need to improve safety requires monitoring personnel to watch the monitors day and night, but this will bring a heavy workload, especially when there are many monitoring points, it is difficult for monitoring personnel to achieve comprehensive monitoring;

[0007] Based on this, the present invention proposes a monitoring image processing system and method based on big data, which can actively analyze monitoring functions and automatically issue warnings to free people from long-term, boring and arduous monitoring operations and improve monitoring effects. Summary of the invention

[0008] The purpose of the present invention is to provide a monitoring image processing system and method based on big data to solve the shortcomings of the background technology.

[0009] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a monitoring image processing system based on big data, comprising an image stitching module, a background model building module, a target detection module, a target tracking module, a feature extraction module and an abnormal warning module;

[0010] Image stitching module: continuously collects several frames of background images without moving objects, and automatically processes several background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images;

[0011] Background model building module: Use N frames of panoramic images to build a background model of the monitoring area based on the mixed Gaussian algorithm;

[0012] Target detection module: When a moving target appears in the background model, the moving target of interest is extracted from the background image through the moving target detection algorithm;

[0013] Target tracking module: Tracks the moving target in the background model through the particle filter tracking algorithm and records the target's walking trajectory;

[0014] Feature extraction module: Use the SIFT feature extraction algorithm to extract SIFT feature points on the image returned by the camera lens, and then use the RANSAC robust fitting algorithm to match the feature points of the panoramic image to obtain matching feature point pairs. Use the homography matrix to project the image, perform a difference operation on the transformed image in the overlapping area, and pre-process the difference result according to the threshold to obtain the binary foreground image of the moving target;

[0015] Abnormal warning module: Perform abnormal analysis on the behavior of the moving target in the binary foreground image. When the moving target is analyzed to have abnormal behavior, a warning is issued to the moving target and a warning message is sent to the management personnel.

[0016] Preferably, the moving target detection algorithm includes an optical flow method, an inter-frame difference method, and a background difference method;

[0017] Optical flow method: assign a velocity vector to each pixel in the image to form an image motion field. The image motion field is the projection of a three-dimensional object on the image plane when it moves. When there is relative motion between an object and the background, the velocity vector formed by the moving object will differ from the background velocity vector of the neighborhood, and the position of the moving object can be detected.

[0018] Inter-frame difference method: compare the grayscale values ​​of corresponding pixels of two frames and calculate the grayscale value difference. The expression is: H k (x,y)=|I(x,y,k)-I(x,y,k-1)|;

[0019] In the formula, H k(x, y) represents the grayscale difference image of the kth frame, I(x, y, k) represents the pixel at the (x, y) coordinate in the k-1th frame image, and I(x, y, k-1) represents the pixel at the (x, y) coordinate in the k-1th frame image. The grayscale difference H of the kth frame image is obtained. k After (x, y), the k-th frame grayscale difference image is binarized to obtain the k-th frame binary image, the expression is:

[0020]

[0021] Where P k (x, y) represents the binary image of frame k obtained by binarizing the gray value difference image of frame k, H k (x, y) represents the gray value difference image of the kth frame, and T1 represents the inter-frame difference threshold for binarizing the image;

[0022] Background difference method: subtract the current image from the background image, and then perform thresholding to detect the moving object area.

[0023] Preferably, the target monitoring module includes an algorithm selection unit, which is used to comprehensively analyze the environmental conditions of the monitoring area and the conditions of the processing equipment, and then determine which moving target detection algorithm to select to detect the target in the monitoring area.

[0024] Preferably, the algorithm selection unit obtains the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index, and sets proportional coefficients for the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index, respectively. The proportional coefficients are used to adjust the numerical values ​​of the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index. After adjusting the numerical values ​​of the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index, the target moving speed is directly proportional to the selection coefficient, and the device computing resource occupancy index and the illumination variation coefficient are inversely proportional to the selection coefficient. The selection coefficient is obtained by comprehensive calculation.

[0025] Preferably, after the algorithm selection unit obtains the selection coefficient value, the obtained selection coefficient xz s After substituting the value into the algorithm matching table, the moving target detection algorithm is selected. The moving target detection algorithms selected in descending order of coefficient values ​​are optical flow method first, frame difference method first, and background difference method first.

[0026] Preferably, the image stitching module continuously collects several frames of background images without moving objects, and automatically processes the several background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images, including the following steps:

[0027] Feature point extraction: Extract image features through SIFT feature point detection operator;

[0028] Feature point matching: After extracting relevant feature points of the image through the SIFT feature point detection operator, the feature points are parameterized and described, and then the parameters of the feature points are used to perform feature point matching;

[0029] Establish a projection model: obtain the perspective change matrix between the images to be stitched based on the mapping relationship between the matching feature point pairs, and then map the images to be stitched into the reference coordinate space;

[0030] Image fusion: Processing the stitching seams in the images in the reference coordinate space caused by the differences in illumination and brightness of the images to be stitched, and achieving smooth and seamless transition between the stitched images.

[0031] Preferably, the background model building module uses N panoramic images to build a background model of the monitoring area based on a mixed Gaussian algorithm, including the following steps:

[0032] A. Initialize the background model: Select the first N frames of panoramic images to initialize the background model. For each pixel position, calculate the mean and variance of the corresponding pixel values ​​in these N frames of images to establish the initial background model. Build a mixed Gaussian model for each pixel, which initially includes Gaussian components to represent the background.

[0033] B. Model update: For each new panoramic image, each pixel is updated to calculate the distance between the current pixel value and the mean and variance of each Gaussian component in the background model at the pixel position. If the distance between the pixel value and a Gaussian component is less than the distance threshold, the pixel is classified as background. If the distance between the pixel value and all Gaussian components is greater than the distance threshold, the pixel is classified as foreground.

[0034] C. Gaussian model maintenance: For pixels classified as background, update the mean and variance of the corresponding Gaussian component. For pixels classified as foreground, add a new Gaussian component to the background model.

[0035] D. Mixed component management: Regularly check the number of Gaussian components in the background model at each pixel position. If the number of Gaussian components exceeds the preset upper limit, choose to merge or delete the components;

[0036] E. Loop iteration: Repeat steps B to D to process consecutive panoramic image frames and continuously update the background model.

[0037] The present invention also provides a monitoring image processing method based on big data, the processing method comprising the following steps:

[0038] S1: The acquisition end continuously acquires several frames of background images without moving objects, and automatically processes several background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images;

[0039] S2: The processing end uses N frames of panoramic images to establish a background model of the monitoring area based on a mixed Gaussian algorithm;

[0040] S3: When a moving target appears in the background model, the moving target is extracted from the background image through the moving target detection algorithm, and the moving target in the background model is tracked through the particle filter tracking algorithm to record the target walking trajectory;

[0041] S4: Using the SIFT feature extraction algorithm to extract SIFT feature points on the image returned by the camera lens, and then using the RANSAC robust fitting algorithm to match the feature points of the panoramic image to obtain matching feature point pairs;

[0042] S5: After obtaining the projection relationship between the images, the homography matrix is ​​used to perform projection transformation on the images, and the transformed images are subjected to differential operation in the overlapping area. The differential results are processed according to the threshold value to obtain the binary foreground image of the moving target;

[0043] S6: Perform abnormal analysis on the behavior of the moving target in the binary foreground image. When the moving target is found to have abnormal behavior, a warning is issued to the moving target, and a warning message is sent to the management personnel.

[0044] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0045] After the present invention generates N large-size seamless panoramic images through the image stitching module, the background model building module uses the N panoramic images to build the background model of the monitoring area based on the mixed Gaussian algorithm. When a moving target appears in the background model, the target detection module extracts the moving target of interest from the background image through the moving target detection algorithm. The target tracking module tracks the moving target in the background model through the particle filter tracking algorithm and records the target walking trajectory. The abnormal warning module performs abnormal analysis on the behavior of the moving target in the binary foreground image. When the moving target is analyzed to have abnormal behavior, a warning is issued to the moving target and a warning message is sent to the management personnel. The processing system can actively analyze the monitoring function and automatically warn to free people from long-term, boring and heavy monitoring operations and improve the monitoring effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0047] Figure 1 It is a system module diagram of the present invention;

[0048] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example 1: Please refer to Figure 1 As shown, the monitoring image processing system based on big data described in this embodiment includes an image stitching module, a background model building module, a target detection module, a target tracking module, a feature extraction module and an abnormal warning module;

[0051] Image stitching module: Continuously collect several frames of background images without moving objects, and automatically match and convert several background images from the same scene with a certain overlapping area based on the image stitching algorithm of SIFT feature points, and finally fuse them to obtain N frames of large-size seamless panoramic images. The panoramic images are used to initialize the background model and are sent to the background model establishment module and feature extraction module.

[0052] Background model building module: N frames of panoramic images are used to build a background model of the monitoring area based on the mixed Gaussian algorithm, and the background model information is sent to the target detection module.

[0053] Target detection module: When a moving target appears in the background model, the moving target of interest is extracted from the background image through the moving target detection algorithm. The moving target detection algorithm includes the optical flow method, the inter-frame difference method and the background difference method. The moving target information is sent to the target tracking module;

[0054] 1) Optical flow method: Usually, a moving target refers to a target that has a certain displacement change with the background, so the target can be segmented by analyzing the velocity vector characteristics of each pixel on the image. The classic algorithm is the optical flow method. The basic principle of the optical flow method to detect moving targets is to assign a velocity vector to each pixel in the image to form an image motion field, which is the projection of the three-dimensional object on the image plane when it moves. What we see is the change of image brightness over time, that is, the optical flow field. If there is no moving object in the image, the optical flow vector changes continuously in the entire image area;

[0055] When there is relative motion between an object and the background, the velocity vector formed by the moving object will be different from the background velocity vector of the neighborhood, so that the position of the moving object can be detected. The advantage of the optical flow method is that even if the camera is moving, it can detect independent moving targets. However, most optical flow algorithms are complex to implement, and have poor noise resistance and real-time performance. They are generally only suitable for identifying rigid moving objects. Without special hardware devices, they are not suitable for real-time processing of video streams.

[0056] 2) Inter-frame difference method: Inter-frame difference, also known as time domain difference (DT), is the simplest method to detect changes between two adjacent frames in an image sequence. If an object at a certain position in an image changes, then the grayscale value of the image corresponding to that position also changes, while the grayscale change of the unchanged part is not obvious. The inter-frame difference method uses this idea. This method directly compares the grayscale values ​​of the corresponding pixels of the two frames of images and calculates the grayscale value difference. The expression is: H k (x,y)=|I(x,y,k)-I(x,y,k-1)|;

[0057] In the formula, H k (x, y) represents the grayscale difference image of the kth frame, I(x, y, k) represents the pixel at the (x, y) coordinate in the k-1th frame image, and I(x, y, k-1) represents the pixel at the (x, y) coordinate in the k-1th frame image. The grayscale difference H of the kth frame image is obtained. k After (x, y), the k-th frame grayscale difference image is binarized to obtain the k-th frame binary image, the expression is:

[0058]

[0059] Where P k (x, y) represents the binary image of frame k obtained by binarizing the gray value difference image of frame k, H k (x, y) represents the grayscale difference image of the kth frame, and T1 represents the inter-frame difference threshold for binarizing the image.

[0060] The advantages of using the inter-frame difference method for moving target detection and extraction are: sensitive to moving objects, simple algorithm implementation, low program design complexity, and easy to achieve real-time monitoring. Its disadvantages include the following: if there are large similar areas inside the moving object, these internal areas will be difficult to detect, thus forming holes. The background area that appears behind the moving object will be mistaken for a moving target because of the difference with the previous frame. In addition, due to the influence of noise, there are many isolated points in the detection results, making it difficult to segment the moving target well.

[0061] 3) Background difference method: Background difference is also called background subtraction, which is a technique that subtracts the current image from the background image and then performs thresholding to detect the moving object area. The detection of moving objects using the background difference method is mainly divided into three steps:

[0062] 3.1) Perform statistical modeling for each pixel in the background;

[0063] 3.2) Compare the current image with the background model and find out the pixels where the current image deviates from the background model by a certain threshold;

[0064] 3.3) The image is binarized to obtain a set of binarized pixels of the moving object. The background difference method is described as follows:

[0065] B(i,j)=|J(i,j)-K(i,j)|;

[0066]

[0067] Where B(i, j) is the pixel difference image, J(i, j) represents the current video image, K(i, j) represents the background image, and T2 represents the background difference threshold for binarizing the image.

[0068] The key to the success of object detection using background difference method lies in whether the model describing the background image, that is, the background model, is accurate. It is the basis of background difference method. A simple and commonly used way to establish a background model is:

[0069] Directly using an image of a non-moving object obtained from the camera as the background, or using the average of multiple images of non-moving objects as the background, can obtain relatively complete background feature data. However, it is particularly sensitive to interference from other external events in the scene, such as light, swaying motion of branches, rotation of fans, etc. Currently, the more classic background model is mainly the mixed Gaussian background model.

[0070] Since the above three moving target detection algorithms have their own characteristics under different requirements and different scenarios, in order to improve the detection efficiency and detection effect of moving targets in corresponding scenarios, we propose the following scheme, which can enable the target detection module to adaptively match the moving target detection algorithm in the scene and improve the detection efficiency and detection effect of moving targets. The details are as follows:

[0071] The target monitoring module also includes an algorithm selection unit, which is used to comprehensively analyze the environmental conditions of the monitoring area and the processing equipment conditions, and then determine which moving target detection algorithm to select to detect the target in the monitoring area;

[0072] When performing moving target detection, obtain detection environment parameters and device parameters;

[0073] Environmental parameters include target movement speed and illumination variation coefficient, and device parameters include device computing resource occupancy index;

[0074] The algorithm selection unit marks the target moving speed, illumination change coefficient, and device computing resource occupancy index as lx y , gz b ,js z ; And generate the selection coefficient xz by comprehensively calculating the target moving speed, illumination change coefficient, and device computing resource occupancy index s ;

[0075] Selection coefficient xz s The calculation expression is: In the formula, α, β, and γ are the proportional coefficients of the illumination change coefficient, the device computing resource occupation index, and the target moving speed, respectively, and α, β, and γ are all greater than 0. The obtained selection coefficient xz s After substituting the value into the algorithm matching table, the moving target detection algorithm is selected. The algorithm matching table is shown in Table 1:

[0076] Moving object detection algorithm <![CDATA[Selection coefficient xz s value]]> Optical flow method 10.226～12.578 Inter-frame difference method 5.512～10.226 Background subtraction method 1～5.513

[0077] Table 1

[0078] As shown in Table 1, the selection coefficient xz s The moving target detection algorithms selected from large to small values ​​are: optical flow method takes precedence over inter-frame difference method takes precedence over background difference method;

[0079] Selection coefficient xz s The calculation logic is:

[0080] When the target detection module detects that a target appears inside the monitoring area, the algorithm selection unit obtains the target movement speed, the illumination change coefficient, and the device computing resource occupancy index, and sets the proportional coefficient for the target movement speed, the illumination change coefficient, and the device computing resource occupancy index respectively. The proportional coefficient is used to adjust the numerical values ​​of the target movement speed, the illumination change coefficient, and the device computing resource occupancy index (to avoid a certain parameter value being too large, resulting in the influence of the other two parameters being too small). After adjusting the numerical values ​​of the target movement speed, the illumination change coefficient, and the device computing resource occupancy index, the target movement speed and the selection coefficient xz s There is a positive relationship between the equipment calculation resource occupancy index, the illumination variation coefficient and the selection coefficient xz s Inversely proportional, the selection coefficient is finally calculated and obtained;

[0081] The target moving speed indicates the moving speed of the target in the monitoring area. The moving target detection algorithms are selected from the largest to the smallest target moving speed, namely, the optical flow method takes precedence over the frame difference method and the background difference method takes precedence over the background difference method. The specific reasons are as follows:

[0082] Optical flow method: Optical flow method is preferred; Optical flow method is a technique suitable for detecting fast-moving targets because it is specifically used to estimate the pixel displacement between consecutive frames and can capture the subtle movement of the target; Optical flow method generally provides good results when the target is fast; however, it is relatively sensitive to noise and lighting changes in the image, so it needs to be applied in conditions with low noise and stable lighting;

[0083] Inter-frame difference method: If the application of the optical flow method is disturbed by illumination changes or noise, or higher real-time performance is required, then the inter-frame difference method can be an alternative; the inter-frame difference method detects the movement of the target by comparing the pixel differences between consecutive frames; it is relatively adaptable to the target speed, but may be less sensitive to the slight movement of the target;

[0084] Background subtraction: Background subtraction is usually not the first choice because it is more suitable for static backgrounds and relatively slow-moving targets; when the target's speed is very high, background subtraction may not be able to capture the target's motion because the target's position change between adjacent frames may exceed its detection ability.

[0085] The calculation expression of the illumination variation coefficient is:

[0086] Where a = {1, 2, 3, ..., n}, n represents the number of monitoring points in the monitoring period, n is a positive integer, P a It represents the light intensity at the ath monitoring point, P represents the average light intensity, and the larger the light variation coefficient is, the greater the light intensity fluctuation in the monitoring area is during the monitoring period, and the greater the light intensity change in the monitoring environment is.

[0087] The moving target detection algorithms are selected from large to small according to the illumination change coefficient. The background difference method takes precedence over the frame difference method and the optical flow method takes precedence. The specific reasons are as follows:

[0088] Background difference method: Background difference method is usually less sensitive to lighting changes because it focuses on detecting relatively static targets. In environments with large changes in lighting conditions, background difference method is more suitable because it mainly focuses on the pixel differences between the target and the background;

[0089] Inter-frame difference method: The inter-frame difference method is usually sensitive to illumination changes. It detects the motion of the target by comparing the pixel differences between consecutive frames, so it can adapt to illumination changes to a certain extent;

[0090] Optical flow method: Optical flow method is usually sensitive to illumination changes because it relies on pixel-level displacement estimation, and illumination changes may cause changes in pixel intensity; therefore, optical flow method is not well suited for situations with large illumination changes.

[0091] The calculation expression of the device computing resource occupancy index is:

[0092] Where b = 1, 2, ..., m, m is the number of device CPU monitoring time points, and m is a positive integer, cpz b Indicates the device CPU usage at b monitoring time points. A larger device computing resource usage index indicates a higher average device CPU usage. This indicates that the device may be running other software or that the device computing performance has declined.

[0093] The moving target detection algorithms selected from the largest to the smallest device computing resource usage index are background difference method, frame difference method, and optical flow method. The specific reasons are as follows:

[0094] Background difference method: Background difference method is usually a lightweight target detection method with relatively low demand for computing resources. It is mainly used to detect relatively static targets without complex pixel-level displacement estimation or comparison between consecutive frames. Therefore, background difference method may be the most appropriate choice when computing resource usage is high;

[0095] Frame difference method: Frame difference method is relatively lightweight. It detects the motion of the target by comparing the pixel differences between consecutive frames. Although it requires some computing resources, it is usually lighter than the optical flow method.

[0096] Optical flow method: Optical flow method usually requires more computing resources because it involves pixel-level displacement estimation, which may not be efficient enough when computing resources are limited. Therefore, optical flow method is usually the last choice unless other methods cannot meet the detection requirements.

[0097] Target tracking module: Tracks the moving target in the background model through the particle filter tracking algorithm, records the target's walking trajectory, and sends the recorded information to the feature extraction module.

[0098] Feature extraction module: The SIFT feature extraction algorithm is used to extract SIFT feature points on the image returned by the camera lens, and then the RANSAC robust fitting algorithm is used to match the feature points of the panoramic image. The matching feature point pairs are refined to obtain accurate matching feature point pairs. After the projection relationship between the images is obtained, the homography matrix is ​​used to project the images, and the transformed images are differentially operated in the overlapping area. Finally, the differential results are binarized according to the threshold and the gradient morphological operation is performed to obtain the binary foreground image of the moving target. The binary foreground image is sent to the abnormal warning module.

[0099] Abnormal warning module: analyze the abnormal behavior of the moving target in the binary foreground image (the abnormal analysis is set by the user in advance. For example, the restricted area is divided in the monitoring area. The entry into the restricted area requires authorization verification (such as pre-entry processing system, fingerprint recognition, face recognition or password recognition, etc.). If the moving target is not authorized to enter the restricted area, it is judged that the moving target has abnormal behavior, or the moving target moves the static objects in the background model, and it is judged that there is no abnormal behavior, etc., which will not be elaborated here. When the moving target is analyzed to have abnormal behavior, a warning is issued to the moving target, and a warning message is sent to the management personnel. Specifically:

[0100] Target detection: First, the system executes a target detection algorithm, such as frame difference or background difference, to obtain a binary foreground image, where the moving target is displayed as a white area on a black background.

[0101] Abnormal analysis rules: Users pre-set abnormal analysis rules in the system. These rules can include prohibited areas, speed limits, durations, object sizes, etc. For example, entry into a prohibited area requires authorization verification, or if a target stays in a prohibited area for more than a specified time, it is considered an abnormal behavior;

[0102] Abnormal behavior detection: The system will perform abnormal behavior detection on the moving targets in the binary foreground image according to the rules set by the user. This may include detecting whether the target has entered a prohibited area or is unauthorized.

[0103] Alert processing: If the system detects abnormal behavior of a target, it triggers alert processing, which usually includes the following steps:

[0104] a. Issue a warning to the target: The system can issue a warning to the target using sound, light, voice prompt or other means to alert the target that its behavior is abnormal and require appropriate action;

[0105] b. Send warning information: The system will send warning information to management or security personnel so that they can take necessary measures. This can be achieved through SMS, email, mobile application notification, etc.;

[0106] c. Record abnormal events: The system usually records the details of abnormal events, including time, location, camera ID, target image, etc., for subsequent analysis and investigation;

[0107] Continuous monitoring: The abnormal warning module will continuously monitor the moving target to ensure that abnormal situations are handled in a timely manner. Once the target's behavior returns to normal or is authorized, the warning will be lifted.

[0108] After the application generates N large-size seamless panoramic images through the image stitching module, the background model building module uses the N panoramic images to build the background model of the monitoring area based on the mixed Gaussian algorithm. When a moving target appears in the background model, the target detection module extracts the moving target of interest from the background image through the moving target detection algorithm. The target tracking module tracks the moving target in the background model through the particle filter tracking algorithm and records the target's walking trajectory. The abnormal warning module performs abnormal analysis on the behavior of the moving target in the binary foreground image. When the moving target is analyzed to have abnormal behavior, a warning is issued to the moving target and a warning message is sent to the management personnel. The processing system can actively analyze the monitoring function and automatically warn to free people from long-term, boring and heavy monitoring operations and improve the monitoring effect.

[0109] Embodiment 2: The image stitching module continuously collects a number of frames of background images without moving objects, and automatically processes the several background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images, including the following steps:

[0110] 1) Feature point extraction: Feature points can abstractly describe the image content. When the image undergoes geometric or radiation changes, the relevant description information of the feature points will not change significantly. Therefore, feature point detection operators and corresponding descriptors can be used to extract image features and positively describe the image. Currently, the more typical feature point extraction operators are mainly Harris operator and SIFT operator, and they have their own characteristics.

[0111] 2) Feature point matching: After extracting the relevant feature points of the image through the feature point extraction operator, the feature points are parameterized and described, and then the feature point descriptors, i.e. the parameters describing the feature points, are used to match the feature points. Currently, the more commonly used algorithms are the kd-tree-based approximate nearest neighbor search algorithm BBF and the random sampling consensus algorithm RANSAC. The former is used to perform initial matching of feature points, and then the latter is used to obtain accurate matching feature point pairs, thereby completing the registration of the images to be spliced;

[0112] 3) Establishing a projection model: Its purpose is to project the images to be stitched onto the reference coordinate space through the projection model. The usual method is to obtain the 8-parameter perspective change matrix between the images to be stitched by matching the mapping relationship between the feature point pairs, and then map the images to be stitched into the specified reference coordinate space;

[0113] 4) Image fusion: Its purpose is to deal with the stitching seams in the images in the reference coordinate space caused by differences in illumination, brightness, etc. of the images to be stitched, that is, to achieve a smooth and seamless transition between the stitched images.

[0114] The background model building module uses N frames of panoramic images to build a background model of the monitoring area based on a mixed Gaussian algorithm, including the following steps:

[0115] A. Initialize the background model: Select the first N panoramic images, where N is usually a small positive integer, to initialize the background model. For each pixel position, calculate the mean and variance of the corresponding pixel values ​​in these N frames to establish an initial background model. Build a mixed Gaussian model for each pixel, which usually includes a Gaussian component to represent the background.

[0116] B. Model update: For each new panoramic image, the following update operation is performed on each pixel: the distance between the current pixel value and the mean and variance of each Gaussian component in the background model at the pixel position is calculated. If the distance between the pixel value and a Gaussian component is less than a distance threshold (usually a threshold of multiple standard deviations), the pixel is classified as the Gaussian component, indicating that it is part of the background. If the distance between the pixel value and all Gaussian components is greater than the distance threshold, it means that the pixel does not belong to the background and should be classified as the foreground.

[0117] C. Gaussian model maintenance: For pixels classified as background, the mean and variance of the corresponding Gaussian component are updated to adapt to the new observation data. For pixels classified as foreground, different strategies can be adopted, such as adding them to a new Gaussian component in the background model, or continuing to track foreground pixels and triggering corresponding alarms;

[0118] D. Mixed component management: Regularly check the number of Gaussian components in the background model at each pixel position. If the number of Gaussian components exceeds the preset upper limit, some components can be merged or deleted to reduce the computational complexity and improve the stability of the model.

[0119] E. Loop iteration: Repeat steps B to D to process consecutive panoramic image frames and continuously update the background model.

[0120] The target tracking module tracks the moving target in the background model through the particle filter tracking algorithm and records the target's walking trajectory, including the following steps:

[0121] 1) Initialization: Randomly generate a set of particles (also called hypotheses) that will be used to estimate the location of the target and build an appearance model of the target, usually the shape, color and other features of the target.

[0122] 2) Prediction: Based on the motion characteristics of the target, a state transition model is used to predict the next position of each particle, and random process noise is introduced to simulate uncertainty;

[0123] 3) Measurement update: Use surveillance cameras or sensors to obtain observation data of the target, usually the appearance features of the target, such as color, texture, etc., compare the predicted position of each particle with the actual observation data, calculate the particle weight, that is, the degree of fit between the particle and the observation data, and normalize the particle weights to ensure that their sum is equal to 1;

[0124] 4) Resampling: According to the weight of the particles, resampling operation is performed. Particles with higher weights will be more likely to be retained, while particles with lower weights will be replaced to make the particle distribution closer to the actual target position;

[0125] 5) Update the target model: Based on the retained particles, update the target’s appearance model to better match the observed data;

[0126] 6) Record target trajectory: Keep an estimate of the target position in each step to record the target's walking trajectory. The target's trajectory is usually represented by the coordinates of the target's center position or other feature points. The trajectory coordinates can be saved in historical data for subsequent analysis and visualization.

[0127] 7) Loop iteration: Repeat steps 2) to 6) to continuously track the target and record the trajectory. The particle filter algorithm will continuously update the observed data and the state transition model to provide the estimated position and trajectory of the target.

[0128] The feature extraction module uses the SIFT feature extraction algorithm to extract SIFT feature points on the image returned by the camera lens, and then uses the RANSAC robust fitting algorithm to match the feature points of the panoramic image to obtain matching feature point pairs. The homography matrix is ​​used to project the image, and the transformed image is differentially operated in the overlapping area. The differential result is preprocessed according to the threshold to obtain the binary foreground image of the moving target, including the following steps:

[0129] The panorama GI (Global-Image) is obtained by image stitching technology, and the SIFT feature points of the panorama are extracted and saved in the set GF (Global-Feature). This step only needs to be performed once when the system is initialized, so that the time for abnormal target detection in the future can be reduced. The other steps need to be performed once for each detection;

[0130] Extract the lens image CI (Camera-Image), that is, obtain the real-time image of the current lens, extract the SIFT feature points of the lens image and save them in the set GF (Camera-Feature);

[0131] The approximate nearest neighbor search algorithm BBF is used to search for matching feature points in CI and GF, and then the random sampling consensus algorithm RANSAC is used to refine the obtained matching feature point pairs, and the perspective transformation matrix H from the lens image to the panoramic image is obtained. The lens image is transformed to the coordinates of the panoramic image to obtain the transformed lens image TCI (Transformed-Camera-Image);

[0132] Set the area where the actual lens image in TCI is located as the region of interest ROI (Region-Of-Interest), and set the same ROI in the panoramic image GI, so that the subsequent operations are only valid within the ROI. Perform the difference operation between TCI and GI, and obtain the binary foreground image BCI (Binary-Camera-Image) of the original lens image after thresholding;

[0133] Due to the existence of noise or deficiencies in the stitching of panoramic images, the binary foreground image will have some noise points. Therefore, morphological filtering is required to determine the area where the intrusion target is located. Finally, the features of the intrusion abnormal target are extracted in TCI to provide a basis for tracking and identification.

[0134] Since the feature points extracted by the SIFT algorithm are highly robust and invariant to image rotation, scaling, translation, light, occlusion, etc., the SIFT feature matching algorithm can be used to align two images.

[0135] After completing the matching of image feature points, before performing the image difference operation, it is also necessary to convert the lens image to a coordinate system consistent with the panoramic image, that is, to find the mapping relationship between the two images through the obtained feature point set. After extracting the matching feature point set in the overlapping area of ​​the lens image and the panoramic image, the perspective matrix is ​​continuously refined through iteration through the feature point set, and then the lens image is transformed to the coordinate system where the panoramic image is located according to the obtained perspective transformation matrix.

[0136] This application is mainly used in small-resolution fixed background monitoring mode and large-resolution PTZ change patrol scanning monitoring mode:

[0137] 1) Small-resolution fixed-background monitoring mode: In the traditional monitoring mode, if the monitoring screen is to include a larger monitoring scene, only a small-resolution screen can be obtained. Even if a larger intrusion target can be detected, the image features of the intrusion target that can be extracted are not obvious due to the small resolution, which greatly increases the difficulty of correctly identifying the target. If the target is to be better identified, it is necessary to obtain an image of the target with a larger resolution, that is, it is necessary to increase the optical magnification of the camera and accurately position the camera to the target area. After identifying the intrusion target, the camera is transformed back to the original resolution for fixed-background monitoring, so as not to affect the monitoring of the entire scene. Based on this idea, the specific workflow of this monitoring mode is as follows: first, in the case of a small-resolution fixed background, use the basic The background difference method of the mixed Gaussian background model is used to detect intrusion targets in the scene. When an intrusion target is found, the lens is positioned to the area where the intrusion target is located and the lens magnification is increased. At this time, because the camera magnification is increased and the pan / tilt is rotated, the background image will undergo changes in scale, rotation, and even brightness. Therefore, the background difference method cannot be used directly to detect the target. The current lens image needs to be aligned to the panoramic image through SIFT features, and then converted to the coordinate system where the panoramic image is located, and then the difference method is used to obtain the abnormal intrusion target image. After extracting the clear features of the target, the camera is adjusted back to the original small resolution fixed background mode, and the particle filter algorithm is used to track the target, save the motion trajectory of the intrusion target, and issue relevant alarms as needed.

[0138] 2) High-resolution pan-tilt motion scanning monitoring mode: In the low-resolution fixed background monitoring mode, some obvious intrusion targets or anomalies in the scene can be found, but some relatively small changes may be ignored as noise, while some key small changes may cause major accidents. For example, branches and plastic bags on some equipment in the substation caused by external forces may cause damage to the equipment and need to be processed in time. Therefore, intelligent monitoring under high-resolution video images is required. In the high-resolution pan-tilt motion scanning mode, the camera uses a large magnification, so that a large-resolution image can be obtained, so that some small changes in the monitoring scene can be found, and the security of the monitoring system can be improved. Because in this mode, the camera pan is rotating and the lens background is not fixed, so at this time it is necessary to perform image matching first. The abnormal target detection algorithm based on SIFT feature matching can meet the requirements. In this monitoring mode, not only moving targets can be detected, but also some abnormal static intrusion targets in the panoramic image can be detected. This monitoring mode can also replace people to check some important monitoring points in sequence along the set path, so as to achieve high security and no guarding.

[0139] Example 3: Please refer to Figure 2 As shown, the monitoring image processing method based on big data described in this embodiment includes the following steps:

[0140] The acquisition end continuously collects several frames of background images without moving objects. The image stitching algorithm based on SIFT feature points automatically matches and converts several background images from the same scene with certain overlapping areas, and finally fuses them to obtain N frames of large-size seamless panoramic images. The panoramic images are used to initialize the background model. The processing end uses N frames of panoramic images to establish a background model of the monitoring area based on the mixed Gaussian algorithm. When a moving target appears in the background model, the moving target of interest is extracted from the background image through the moving target detection algorithm, and the moving target in the background model is tracked through the particle filter tracking algorithm. The target walking trajectory is recorded, and the SIFT feature extraction algorithm is used to extract SIFT feature points on the image returned by the camera lens, and then the RANSAC robust fitting algorithm is used to match them with the feature points of the panoramic image, and the matching feature point pairs are refined to obtain accurate The matching feature point pairs are obtained. After the projection relationship between the images is obtained, the homography matrix is ​​used to perform projection transformation on the images. The transformed images are differentially operated in the overlapping area. Finally, the differential results are binarized according to the threshold and gradient morphological operations are performed to obtain the binary foreground image of the moving target. The behavior of the moving target in the binary foreground image is analyzed for abnormalities (the abnormal analysis is set by the user in advance. For example, a restricted area is divided in the monitoring area, and the entry into the restricted area requires authorization verification (such as pre-entry processing system, fingerprint recognition, face recognition or password recognition, etc.)). If the moving target is not authorized to enter the restricted area, it is judged that the moving target has abnormal behavior, or when the moving target moves the static objects in the background model, it is judged that there is no abnormal behavior, etc., which will not be elaborated here. When it is analyzed that the moving target has abnormal behavior, a warning is issued to the moving target, and a warning message is sent to the management personnel.

[0141] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0142] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0143] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0144] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0145] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0146] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A monitoring image processing system based on big data, characterized in that: It includes image stitching module, background model building module, target detection module, target tracking module, feature extraction module and abnormal warning module; Image stitching module: continuously collects several frames of background images without moving objects, and automatically processes several background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images; Background model building module: Use N frames of panoramic images to build a background model of the monitoring area based on the mixed Gaussian algorithm; Target detection module: When a moving target appears in the background model, the moving target of interest is extracted from the background image through the moving target detection algorithm; Target tracking module: Tracks the moving target in the background model through the particle filter tracking algorithm and records the target's walking trajectory; Feature extraction module: Use the SIFT feature extraction algorithm to extract SIFT feature points on the image returned by the camera lens, and then use the RANSAC robust fitting algorithm to match the feature points of the panoramic image to obtain matching feature point pairs. Use the homography matrix to project the image, perform a difference operation on the transformed image in the overlapping area, and pre-process the difference result according to the threshold to obtain the binary foreground image of the moving target; Abnormal warning module: Perform abnormal analysis on the behavior of the moving target in the binary foreground image. When the moving target is analyzed to have abnormal behavior, a warning is issued to the moving target and a warning message is sent to the management personnel.

2. The monitoring image processing system based on big data according to claim 1, characterized in that: The moving target detection algorithm includes an optical flow method, an inter-frame difference method, and a background difference method; Optical flow method: assign a velocity vector to each pixel in the image to form an image motion field. The image motion field is the projection of a three-dimensional object on the image plane when it moves. When there is relative motion between an object and the background, the velocity vector formed by the moving object will differ from the background velocity vector of the neighborhood, and the position of the moving object can be detected. Inter-frame difference method: compare the grayscale values ​​of corresponding pixels of two frames and calculate the grayscale value difference. The expression is: H k (x,y)=|I(x,y,k)-I(x,y,k-1)|; In the formula, H k (x, y) represents the grayscale difference image of the kth frame, I(x, y, k) represents the pixel at the (x, y) coordinate in the k-1th frame image, and I(x, y, k-1) represents the pixel at the (x, y) coordinate in the k-1th frame image. Get the grayscale difference H of the kth frame image k (x, y), the k-th frame grayscale difference image is binarized to obtain the k-th frame binary image, the expression is: Where P k (x, y) represents the binary image of frame k obtained by binarizing the grayscale difference image of frame k, H k (x, y) represents the gray value difference image of the kth frame, and T1 represents the inter-frame difference threshold for binarizing the image; Background difference method: subtract the current image from the background image, and then perform thresholding to detect the moving object area.

3. The monitoring image processing system based on big data according to claim 2 is characterized in that: The target monitoring module includes an algorithm selection unit, which is used to comprehensively analyze the environmental conditions of the monitoring area and the conditions of the processing equipment, and then determine which moving target detection algorithm to select to detect the target in the monitoring area.

4. The monitoring image processing system based on big data according to claim 3 is characterized in that: The algorithm selection unit obtains the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index, and sets proportional coefficients for the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index, respectively. The proportional coefficients are used to adjust the numerical values ​​of the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index. After adjusting the numerical values ​​of the target moving speed, the illumination variation coefficient, and the device computing resource occupancy index, the target moving speed is directly proportional to the selection coefficient, and the device computing resource occupancy index and the illumination variation coefficient are inversely proportional to the selection coefficient. The selection coefficient is obtained by comprehensive calculation.

5. The monitoring image processing system based on big data according to claim 4 is characterized in that: After the algorithm selection unit obtains the selection coefficient value, the obtained selection coefficient xz s After substituting the value into the algorithm matching table, the moving target detection algorithm is selected. The moving target detection algorithms selected in descending order of coefficient values ​​are optical flow method first, frame difference method first, and background difference method first.

6. The monitoring image processing system based on big data according to claim 5, characterized in that: The image stitching module continuously collects a number of background images without moving objects, and automatically processes the background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images, including the following steps: Feature point extraction: Extract image features through SIFT feature point detection operator; Feature point matching: After extracting relevant feature points of the image through the SIFT feature point detection operator, the feature points are parameterized and described, and then the parameters of the feature points are used to perform feature point matching; Establish a projection model: obtain the perspective change matrix between the images to be stitched based on the mapping relationship between the matching feature point pairs, and then map the images to be stitched into the reference coordinate space; Image fusion: Processing the stitching seams in the images in the reference coordinate space caused by the differences in illumination and brightness of the images to be stitched, and achieving smooth and seamless transition between the stitched images.

7. The monitoring image processing system based on big data according to claim 6 is characterized in that: The background model building module uses N frames of panoramic images to build a background model of the monitoring area based on a mixed Gaussian algorithm, including the following steps: A. Initialize the background model: Select the first N frames of panoramic images to initialize the background model. For each pixel position, calculate the mean and variance of the corresponding pixel values ​​in these N frames of images to establish the initial background model. Build a mixed Gaussian model for each pixel, which initially includes Gaussian components to represent the background. B. Model update: For each new panoramic image, each pixel is updated to calculate the distance between the current pixel value and the mean and variance of each Gaussian component in the background model at the pixel position. If the distance between the pixel value and a Gaussian component is less than the distance threshold, the pixel is classified as background. If the distance between the pixel value and all Gaussian components is greater than the distance threshold, the pixel is classified as foreground. C. Gaussian model maintenance: For pixels classified as background, update the mean and variance of the corresponding Gaussian component. For pixels classified as foreground, add a new Gaussian component to the background model. D. Mixed component management: Regularly check the number of Gaussian components in the background model at each pixel position. If the number of Gaussian components exceeds the preset upper limit, choose to merge or delete the components; E. Loop iteration: Repeat steps B to D to process consecutive panoramic image frames and continuously update the background model.

8. A monitoring image processing method based on big data, implemented by the processing system according to any one of claims 1 to 7, characterized in that: The processing method comprises the following steps: S1: The acquisition end continuously acquires several frames of background images without moving objects, and automatically processes several background images based on the image stitching algorithm of SIFT feature points to obtain N frames of panoramic images; S2: The processing end uses N frames of panoramic images to establish a background model of the monitoring area based on a mixed Gaussian algorithm; S3: When a moving target appears in the background model, the moving target is extracted from the background image through the moving target detection algorithm, and the moving target in the background model is tracked through the particle filter tracking algorithm to record the target walking trajectory; S4: Using the SIFT feature extraction algorithm to extract SIFT feature points on the image returned by the camera lens, and then using the RANSAC robust fitting algorithm to match the feature points of the panoramic image to obtain matching feature point pairs; S5: After obtaining the projection relationship between the images, the homography matrix is ​​used to perform projection transformation on the images, and the transformed images are subjected to differential operation in the overlapping area. The differential results are processed according to the threshold value to obtain the binary foreground image of the moving target; S6: Perform abnormal analysis on the behavior of the moving target in the binary foreground image. When the moving target is found to have abnormal behavior, a warning is issued to the moving target, and a warning message is sent to the management personnel.

Images