Moving object detection method and system based on optical flow estimation and GMM algorithm

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a moving object detection method based on optical flow estimation and GMM (Gaussian Mixture Model), which comprises the following steps of: 1, accurately compensating a background model; step 2, final segmentation of the foreground image; 3, designing a hardware architecture of the system; according to the method, firstly, the Brox optical flow estimation algorithm and the GMM algorithm are adopted to estimate the flow vector and the background model of each pixel, the flow vectors are estimated more accurately, and the robustness under the condition of brightness change is enhanced; secondly, Brox optical flow estimation is carried out between the compensated background and the current frame, and the flow vectors of all pixels are extracted to reduce the number of false alarms and noise generation; finally, the hardware architecture of the invention comprises an optical flow estimator, a camera motion estimator, a background detector and an object detector. Compared with an existing MOD algorithm, the moving object detector provided by the invention has the advantages that higher precision is realized, real-time processing of 60fps can be supported, and the working frequency is 200MHz.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a moving object detection method and system based on optical flow estimation and GMM algorithm. Background Art

[0002] Most existing moving object detection algorithms assume that the camera is stationary, which limits their effectiveness in practical applications. In advanced driver assistance systems, due to the use of motion cameras, the image frames contain mixed motion of background and objects, which makes it difficult to classify the target object and background. Existing solutions either rely on complex statistical models or outlier detection techniques, or use deep neural networks, but these methods have limitations in terms of power consumption and computing resources. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] The present invention proposes a moving object detection method and system based on optical flow estimation and GMM algorithm, and designs a corresponding hardware architecture to realize real-time processing. The algorithm in the system can effectively perform background compensation and object detection in a moving camera environment, solving the problems raised in the above-mentioned background technology.

[0005] (II) Technical solution

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] A moving object detection method based on optical flow estimation and GMM algorithm comprises the following steps:

[0008] Step 1: Accurately compensate the background model:

[0009] The step 1 specifically includes:

[0010] 1.1) Apply the Brox optical flow estimation algorithm to two consecutive frames of images to extract the motion vector between them;

[0011] 1.2) Using the extracted inter-frame motion data, fine-tune and compensate the previously generated background model;

[0012] 1.3) Adopt GMM algorithm, update and optimize GMM parameters to construct the final background model.

[0013] Step 2: Achieve accurate segmentation of foreground image:

[0014] The step 2 specifically includes:

[0015] 2.1) Use the proposed MOD algorithm to perform background subtraction processing;

[0016] 2.2) Perform Brox optical flow estimation between the compensated background and the current frame;

[0017] 2.3) Apply a median filter to the obtained results to remove smaller interferents.

[0018] Step 3: Design of system hardware architecture:

[0019] The step 3 specifically includes:

[0020] 3.1) Select the pixel intensity of two consecutive frames captured by the external camera module to estimate the flow vector and derive the self-motion through the optical flow estimator;

[0021] 3.2) Perform in-depth analysis of the histogram statistics of these flow vectors using the camera motion estimator;

[0022] 3.3) According to the estimated self-motion displacement, the GMM parameters are adjusted through the background detector;

[0023] 3.4) Apply the object detector to perform background subtraction to determine the exact coordinates of the object.

[0024] Furthermore, in step 1, applying the Brox optical flow estimation algorithm to two consecutive frames of images to extract the motion vector therebetween also includes:

[0025] The flow vector extracted for each pixel reflects its displacement between consecutive frames. Compared with the target area, the pixels in the background area show slower motion characteristics. At the same time, the background pixels occupy a wider area in the image frame and show a consistent motion pattern.

[0026] Furthermore, performing Brox optical flow estimation between the compensated background and the current frame in step 2 further includes:

[0027] Brox optical flow estimation is performed between the compensated background and the current frame to capture the flow vectors of all pixels. Through analysis, those flow vectors that are significantly different from the background in size and direction are classified as target objects.

[0028] Furthermore, in step 3, deducing the self-motion by the optical flow estimator further includes:

[0029] The image frame data stream from the external camera module is stored in the input frame buffers. Subsequently, two consecutive frames are selected from these buffers and their pixel intensity data are used to estimate the flow vectors, and finally the self-motion is inferred by the optical flow estimator.

[0030] A moving object detection system based on optical flow estimation and GMM algorithm, comprising:

[0031] Image acquisition module: focuses on capturing video image sequences from cameras or other video sources in real time; ensuring the high quality and continuity of images, thus laying a solid data foundation for subsequent processing steps;

[0032] Background model management module: responsible for building, updating and maintaining the background model to ensure that it can accurately reflect the environmental conditions of the current scene;

[0033] Optical flow calculation module: The main task is to calculate the optical flow field in the image sequence and estimate the motion information of the object in the image by analyzing the continuous input image frames;

[0034] Self-motion estimation module: dedicated to estimating the self-motion of the camera, and making appropriate compensation adjustments to the background model by analyzing the statistical characteristics of the optical flow field;

[0035] Foreground image segmentation and post-processing module: responsible for separating foreground objects from the background, identifying moving targets in the scene, and performing necessary post-processing;

[0036] Target recognition module: focuses on identifying moving targets in the scene, provides detailed target information, and identifies potential target objects based on the results of foreground extraction.

[0037] (III) Beneficial effects

[0038] Compared with the prior art, the present invention provides a moving object detection method based on optical flow estimation and GMM algorithm, which has the following beneficial effects:

[0039] The present invention firstly adopts Brox optical flow estimation algorithm and GMM algorithm to estimate the flow vector of each pixel and the background model, estimates the flow vector more accurately, and enhances the robustness under the condition of brightness change; secondly, Brox optical flow estimation is performed between the compensated background and the current frame, and the flow vectors of all pixels are extracted to reduce the number of false alarms and the generation of noise; finally, the hardware architecture of the present invention includes an optical flow estimator, a camera motion estimator, a background detector and an object detector, which are implemented on an FPGA device to enhance the real-time performance of target detection.

[0040] The moving object detector proposed in the present invention achieves higher accuracy than the existing MOD algorithm, and can support real-time processing of 60fps with an operating frequency of 200MHz. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Design a flow chart for the system of the present invention;

[0042] Figure 2 It is a general scheme block diagram of the MOD algorithm of the present invention;

[0043] Figure 3 A block diagram of a moving target detector of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Example

[0046] like Figure 1-3 As shown, an embodiment of the present invention proposes a moving object detection method based on optical flow estimation and GMM algorithm, the method comprising the following steps:

[0047] S1. Since the image frames obtained from the motion camera contain the motion of the background and the object, the proposed MOD algorithm performs a background compensation process before detecting the object. To this end, we apply the Brox optical flow estimation algorithm to extract the flow vectors for two consecutive frames. Since the matrix inversion of A in the optical flow estimation requires a computational time proportional to the image frame size, and A is a Hermitian positive definite matrix, we apply the Cholesky decomposition to A:

[0048] A=LL T

[0049] where L is a lower triangular matrix, L T is the transpose of L. The inverse of the original matrix can be calculated by simply multiplying the two inverses, as follows:

[0050] A -1 =(L -1 ) T (L -1 )

[0051] Then, the final flow vector w is calculated as

[0052] w=(L -1 ) T (L -1 )b

[0053] The extracted flow vector for each pixel represents the motion of the pixel between consecutive frames. Pixels in the background area have relatively slow motion compared to the target area. In addition, background pixels occupy a larger area than the object in the image frame and exhibit similar motion. Therefore, the most common slow-motion flow vectors are regarded as self-motion and can be extracted by histogram analysis. The extracted inter-frame motion is then used to compensate the previously derived background model. This method derives the background model through the GMM algorithm and stores it as GMM parameters.

[0054] After estimating the self-motion, the entire GMM parameter can be moved backward along the determined motion to obtain a compensated background model. First, the inter-frame motion e in the x-axis and y-axis directions are respectively dx and e dy Divide into integer part and decimal part. Integer part e i dx and e i dy You can use e dx and e dy Rounding off gives us f dx and e f dy You can also use e dx and e dy Rounding off gives,

[0055]

[0056] Then, the fractional part of the inter-frame motion value, e f dx and e f dy , interpolate the GMM parameters as shown below, where w n,t (x, y) is the weight value of the pixel with coordinates (x, y) at time t, μ n,t (x, y) is the mean value of the pixel coordinates (x, y) at time t, is the variance of the pixel coordinates (x, y) at time t, w n,t-1 (x, y) is the weight value of the pixel with coordinates (x, y) at time t-1, μ n,t-1 (x, y) is the mean value of the pixel coordinates (x, y) at time t-1, is the variance of the pixel coordinates (x, y) at time t-1, is the updated weight value of the pixel with coordinates (x, y) at time t, is the updated mean value of the pixel coordinates (x, y) at time t, is the updated variance of the pixel coordinates (x, y) at time t.

[0057]

[0058] After the compensation process is completed, the GMM parameters updated by the GMM algorithm are used to generate the final background model, that is, the compensated background model.

[0059] S2. In order to extract the target coordinates in the image frame using the compensated background model, the MOD algorithm proposed in this paper first performs a background subtraction operation and then performs Brox optical flow estimation. Background subtraction involves separating moving objects from static background images. If the difference between the compensated background model and the current frame is greater than a threshold, it is classified as a moving object and the rest is classified as background. Although this method can effectively detect objects, such a simple comparison can lead to false positives.

[0060] To address this problem, Brox optical flow estimation is performed between the compensated background and the current frame, and the flow vectors of all pixels are extracted. The extracted flow vectors that are different in size and direction from the background can be grouped into objects. In order to group the target area into one frame from the overall vector, a suitable threshold needs to be determined. This threshold must be selected for each frame by considering the distribution of the flow vectors. Since we derive the detection results by background subtraction, the final detection result is determined by cross-checking with the two sets of results to reduce the number of false positives. A median filter is applied on the results to remove relatively small objects, such as those caused by noise.

[0061] S3. This paper proposes a hardware architecture for a moving object detector for real-time processing. Figure 3 shows the block diagram of the proposed moving object detector, which consists of an optical flow estimator, a camera motion estimator, a background detector, and an object detector. The data stream of image frames coming in from the external camera module is stored in the input frame buffer. Then, two consecutive frames i are selected from these buffers. t-1 and i t The pixel intensity is used to estimate the flow vector u e and v e , and then estimate the self-motion through the optical flow estimator. The histogram statistics of these flow vectors are analyzed by the camera motion estimator to extract the inter-frame motion e dx and e dy In order to generate the compensation background B t The background detector updates the corresponding parameters based on the estimated self-motion displacement GMM parameters by applying the GMM algorithm and using the compensated background B t The target detector performs background subtraction with the pixel intensity of the current frame, and the optical flow estimator simultaneously extracts a new flow vector uo and v o These flow vectors are used by the object detector to classify the target region. Finally, the two sets of detection results are combined to generate the target coordinates.

[0062] In some embodiments, the description of the moving object detector is designed using a hardware description language (HDL) and implemented on a Xilinx Virtex5 FPGA device. The proposed moving object detector is implemented using 13.2K logic slices, 104 dsp48s, and 163 BRAMs.

[0063] Since the final object coordinates are generated at 6.67M clock cycle intervals and the image resolution is 640x480, we confirm that real-time processing of 60fps can be achieved using a 200MHz FPGA test system. The total number of clock cycles is proportional to the resolution of the input image. The results confirm that the proposed system is significantly faster in processing speed (fps) than other solutions that can support real-time processing.

[0064] A moving object detection system based on optical flow estimation and GMM algorithm, comprising:

[0065] Image acquisition module: The image acquisition module is the starting point of the entire visual processing system. It is responsible for capturing video image sequences from cameras or other video sources in real time. The core task of this module is to ensure the quality and continuity of the acquired images, thereby providing a stable and reliable data basis for subsequent image processing and analysis. In order to achieve high-quality video acquisition, the module needs to have high-resolution imaging capabilities, stable frame rate control, and adaptability to different lighting and environmental conditions.

[0066] Background model management module: The background model management module plays a vital role in the visual system. It is responsible for building, continuously updating, and maintaining an accurate background model to ensure that the model can reflect the environmental changes of the current scene in real time. This module analyzes continuous video frames through complex algorithms, identifies and separates background elements, while ignoring dynamic objects in the foreground. This is crucial for subsequent foreground extraction and target recognition.

[0067] Optical flow calculation module: The optical flow calculation module is a key part in processing motion information in image sequences. It receives continuous image frames as input and estimates the motion information of objects in the image through advanced computer vision technology. This information includes not only the speed and direction of the object, but also the shape and size changes of the object. Accurate calculation of optical flow information is crucial to understanding dynamic behavior in the scene.

[0068] Ego-motion estimation module: The ego-motion estimation module focuses on analyzing and estimating the camera's own motion. By deeply analyzing the statistical features in the optical flow field, the module is able to detect the camera's motion and make necessary compensations to the background model accordingly. This compensation mechanism ensures that the background model remains accurate even in the case of camera motion, thereby improving the robustness and accuracy of the entire system.

[0069] Foreground image segmentation and post-processing module: The main task of the foreground image segmentation and post-processing module is to accurately extract foreground objects from the background and identify moving objects in the scene. This module uses advanced image segmentation technology to effectively separate foreground objects from the background and optimize the segmentation results through a series of post-processing operations (such as denoising, morphological processing, etc.). This step is crucial for subsequent target recognition and analysis.

[0070] Target recognition module: The target recognition module is an advanced stage of the visual processing system. It is responsible for further identifying specific moving targets in the identified foreground. This module combines deep learning and other advanced machine learning technologies to perform detailed feature analysis and classification of foreground objects, thereby providing detailed information on the target. Based on the results of foreground extraction, this module can accurately identify potential target objects and provide strong support for subsequent decisions and actions.

[0071] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A moving object detection method based on optical flow estimation and GMM algorithm, characterized in that: The following steps are involved: Step 1: Accurate compensation of background model; The step 1 specifically includes: 1.1) Apply the Brox optical flow estimation algorithm to extract flow vectors for two consecutive frames; 1.2) Using the extracted inter-frame motion value to compensate the previously derived background model; 1.3) Update new GMM parameters through the GMM algorithm to generate the final background model; Step 2: Final segmentation of foreground image: The step 2 specifically includes: 2.1) Perform background subtraction operation according to the proposed MOD algorithm; 2.2) Perform Brox optical flow estimation between the compensated background and the current frame; 2.3) Apply a median filter on the result to remove relatively small objects. Step 3: System hardware architecture design: The step 3 specifically includes: 3.1) Select the pixel intensities of two consecutive frames from the image frames input by the external camera module to estimate the flow vector, and then estimate the self-motion through the optical flow estimator; 3.2) Analyze the histogram statistics of these flow vectors through the camera motion estimator; 3.3) Using the background detector to displace the GMM parameters according to the estimated self-motion; 3.4) Use the target detector to perform background subtraction and generate target coordinates.

2. The moving object detection method based on optical flow estimation and GMM algorithm according to claim 1, characterized in that: The step 3 of estimating the self-motion by the optical flow estimator further includes: The data stream of image frames coming in from the external camera module is stored in input frame buffers. Then, the pixel intensities between two consecutive frames are selected from these buffers to estimate the flow vectors and thus the ego-motion through the optical flow estimator.

3. The moving object detection method based on optical flow estimation and GMM algorithm according to claim 1, characterized in that: In step 2, performing Brox optical flow estimation between the compensated background and the current frame further includes: Brox optical flow estimation is performed between the compensated background and the current frame, and flow vectors of all pixels are extracted. The extracted flow vectors with different sizes and directions from the background can be grouped into objects.

4. The moving object detection method based on optical flow estimation and GMM algorithm according to claim 1, characterized in that: The step 1 of applying the Brox optical flow estimation algorithm to extract flow vectors from two consecutive frames further includes: The extracted flow vector of each pixel represents the motion of the pixel between consecutive frames. Pixels in the background area have relatively slow motion compared to the target area. In addition, background pixels occupy a larger area than objects in the image frame and exhibit similar motion.

5. A moving object detection system based on optical flow estimation and GMM algorithm, characterized in that: include: Image acquisition module: responsible for acquiring video image sequences from cameras or other video sources in real time; ensuring the quality and continuity of images, and providing a reliable data basis for subsequent processing; Background model management module: This module is responsible for the construction, update and maintenance of the background model to ensure that the background model can accurately reflect the environment of the current scene; Optical flow calculation module: This module is responsible for calculating the optical flow field in the image sequence, receiving the input continuous image frames to estimate the motion information of the objects in the image; Self-motion estimation module: This module is responsible for estimating the self-motion of the camera and analyzing the statistical characteristics in the optical flow field in order to compensate the background model accordingly; Foreground image segmentation and post-processing module: This module is responsible for extracting foreground objects from the background, identifying moving objects in the scene, and performing necessary post-processing operations; Target recognition module: This module is responsible for identifying moving targets in the scene, providing detailed information about the target, and identifying potential target objects based on foreground extraction results; Applied to the moving object detection method based on optical flow estimation and GMM algorithm as described in any one of claims 1 to 4.

Images