Discrete picture sequence foreground detection method based on background difference

Abstract

The invention relates to the technical field of image processing, in particular to a background difference-based discrete picture sequence foreground detection method, which comprises the following steps of: photographing and sampling an assembly line to obtain a picture sequence, establishing a background model and initializing; using the initialized background model to carry out front and back background coarse blocking and secondary coarse blocking on the picture sequence to obtain blocked pictures; according to the method, a relatively stable background is obtained through a fixed-interval fixed-point photographing mode, then a background model is established, it is ensured that a foreground target appearing in the moving background can be recognized and detected, and the efficiency and reliability of industrial automation are improved. Meanwhile, for factors such as sudden illumination change and object shielding in a pipeline environment, object movement speed variation and the like are detected, a background model is established through a discrete picture sequence, and foreground target extraction is carried out in cooperation with deep learning, so that the influence can be effectively reduced, and the detection effect and robustness of the method are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a foreground detection method for discrete picture sequences based on background difference. Background Art

[0002] Traditional background difference methods are mainly used to process continuous video frames and can effectively perform target tracking and target detection in relatively stable dynamic scenes. In an industrial assembly line, the background usually changes dynamically and may be continuously adjusted due to the operation of equipment, the movement of products, or environmental changes.

[0003] However, in such a case, it is difficult for traditional background difference methods to accurately extract foreground targets and false detections are likely to occur. Summary of the Invention

[0004] The purpose of the present invention is to provide a foreground detection method for discrete picture sequences based on background difference, aiming to solve the problem that traditional background difference methods are difficult to accurately extract foreground targets and are prone to false detections.

[0005] To achieve the above purpose, the present invention provides a foreground detection method for discrete picture sequences based on background difference, including the following steps:

[0006] Taking pictures of the assembly line for sampling to obtain a picture sequence, establishing a background model and initializing it;

[0007] Using the initialized background model to perform rough foreground and background block division and secondary rough block division on the picture sequence to obtain block pictures;

[0008] Performing fine segmentation on the block pictures, eliminating false foregrounds, and updating the background.

[0009] Among them, the specific method for taking pictures of the assembly line for sampling to obtain a picture sequence, establishing a background model and initializing it:

[0010] Building an industrial camera photographing structure platform and selecting a time interval for fixed photographing sampling to obtain a picture sequence;

[0011] Establishing a background model through discrete pictures and initializing and subsequently updating the background model.

[0012] Among them, the specific method for using the initialized background model to perform rough foreground and background block division and secondary rough block division on the picture sequence to obtain block pictures:

[0013] Using the initialized background model to perform difference on the currently input picture sequence to obtain a difference image;

[0014] The differential image is segmented, and the mean and variance of each segment are calculated respectively. The foreground and the foreground and background are roughly divided by a threshold;

[0015] The foreground and background are roughly segmented again to obtain segmented pictures.

[0016] Among them, the fine segmentation of the segmented pictures, the elimination of false foregrounds, and the update of the background also include that when there is a foreground, the segmented pictures are inferred using a deep learning framework, and the intersection over union of the rectangular frames is calculated together with the results of the traditional method. When the calculation result reaches the threshold, it is recognized as a true foreground.

[0017] Among them, the background is updated according to the variance for the weight factor, and the pictures are stored in an array with a fixed length.

[0018] The foreground detection method for discrete picture sequences based on background difference of the present invention obtains a picture sequence by sampling through pipeline photographing, establishes a background model and initializes it; uses the initialized background model to roughly segment the foreground and background and segment them again roughly for the picture sequence to obtain segmented pictures; performs fine segmentation on the segmented pictures, eliminates false foregrounds, and updates the background. This method obtains a relatively stable background through fixed-interval fixed-point photographing, and then establishes a background model to ensure that foreground targets appearing in a moving background can be recognized and detected, improving the efficiency and reliability of industrial automation. At the same time, in the face of factors such as sudden changes in light, object occlusion, and variable moving speeds of detected objects in the pipeline environment, establishing a background model through discrete picture sequences and combining deep learning for foreground target extraction can effectively reduce its influence, significantly improving the detection effect and robustness of the method. Beneficial effects: The foreground detection method for discrete picture sequences can be applied to the complex and changeable environment of industrial pipelines, solve the target monitoring task in a fixed environment, reduce the labor cost in the production process and improve the monitoring efficiency, and can realize the monitoring and control of product quality in the production process, promoting the further development of industrial automation towards intelligence, and solving the problem that it is difficult for traditional background difference methods to accurately extract foreground targets and easy to cause false detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 is a schematic diagram of the background difference method.

[0021] Figure 2 is a schematic diagram of photographing and modeling.

[0022] Figure 3 It is a schematic diagram of rough segmentation.

[0023] Figure 4 It is a schematic diagram of the process of eliminating pseudo foreground in rough segmentation - fine segmentation.

[0024] Figure 5 It is a schematic diagram of deep learning re - inspection.

[0025] Figure 6 It is a flowchart of the foreground detection method for discrete image sequences based on background difference provided by the present invention.

[0026] Figure 7 It is a flowchart of the specific method for obtaining an image sequence by pipeline photographing sampling, establishing a background model and initializing it.

[0027] Figure 8 It is a flowchart of the specific method for performing rough block division of foreground and background and secondary rough block division on the image sequence by using the initialized background model to obtain block images. Detailed implementation manners

[0028] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] Please refer to Figures 1 to 8 , the present invention provides a foreground detection method for discrete image sequences based on background difference, including the following steps:

[0030] S1 Obtain an image sequence by pipeline photographing sampling, establish a background model and initialize it;

[0031] In the embodiments of the present invention, first, an industrial camera photographing structure platform is built, and a suitable time interval is selected for fixed photographing sampling to ensure obtaining a relatively stable image sequence, so as to initialize and subsequently update the background model. The background model established by discrete images can reduce the influence under complex dynamic backgrounds, improve the efficiency of the method, and reduce the method complexity;

[0032] Specifically, the core of the background difference method is to establish a correct background model, and detect foreground objects by differentiating the current frame from the background model. Under ideal circumstances, as Figure 1 shown, the background template established according to certain rules is subtracted pixel - by - pixel from the current image to obtain the pixel region with a large difference between the current image and the background template, that is, the foreground object.

[0033] The process can be expressed by the following formula (1):

[0034]

[0035] In the formula, the pixel coordinates of the image are represented by (x, y), I'(x, y) is the difference result, abs(M(x, y)-I(x, y)) is the absolute value of the subtraction of the pixels of the template image M(x, y) and the current image I(x, y), and H t is the set threshold;

[0036] Build a background model; as Figure 2 shown, the camera takes a fixed-point photo of the scene at a fixed moment t in a signal-controlled manner, and obtains a sequence of N pictures T{t}={T 0 ,T 1 ,…,T N} at the position at moment t. The template image M is established and updated in real time by taking the average value and the upper and lower threshold templates; as follows

[0037] shown in Table 1:

[0038]

[0039] Table 1 Template Image Establishment Rules

[0040] Among them, M 1 is the mean template, and M h ,M l are the upper and lower threshold templates. After the difference template is established, the difference is performed and then the next step is carried out. The establishment of the background model is very important and plays a key role in the subsequent foreground detection and model update.

[0041] Specific method:

[0042] S11 Build an industrial camera photographing structure platform, and select a time interval for fixed photographing sampling to obtain a picture sequence;

[0043] S12 Establish a background model through discrete pictures, and initialize and subsequently update the background model.

[0044] S2 Use the initialized background model to perform rough foreground and background block division and secondary rough block division on the picture sequence to obtain block pictures;

[0045] In the embodiment of the present invention, after the background model is initialized, for the current input picture sequence X t, perform differencing with the background model to obtain its difference image. The difference image serves as the basis for roughly separating the foreground and background. By dividing the difference image into blocks of size M×N and calculating the mean and variance of each block respectively, the foreground and background are roughly divided by a threshold. There are two types of foreground blocks obtained through rough block division: blocks containing a mixture of foreground and background, and foreground blocks, which need to be further subdivided to further divide the foreground and background, which can effectively improve the method efficiency and speed up the method operation speed;

[0046] Specifically, in order to reduce the method complexity and improve the method operation efficiency, it is necessary to perform rough segmentation on the difference image according to certain rules to initially locate the approximate position where the foreground target appears, and then perform pixel-level fine segmentation on the roughly segmented image. Compared with the traditional background differencing method that calculates pixel by pixel, by locating through rough block division and then segmenting, the method efficiency can be greatly accelerated. For images of the same size (1000*1000) through experiments, the traditional background differencing takes about 90ms, while through rough block division, it can be increased by 3 - 4 times. As follows Figure 3 As shown, perform differencing between the current image and the background image through equation (1), set H t to 0, and the obtained differencing result is D t , and then the differencing result image of size W*H;

[0047] is divided into blocks of size w*h

[0048] Then calculate the pixel mean mean and variance std of each block, and at the same time calculate the variance of the entire image as the threshold H 2 , take the smallest mean ε and variance σ in the four corner blocks of the image as the standard block, and classify the blocks according to formula (5) into three categories: 1. background block; 2. background block containing foreground; 3. foreground block:

[0049]

[0050] Among them, H 1 is set to 3*δ according to the 3δ rule of statistics, which can effectively detect the blocks with foreground. At this time, it is also necessary to perform another rough block division on the image that has been roughly block divided for the first time to perform a more detailed rough segmentation on the block division result, further reducing the method complexity, narrowing the positioning area, and improving the method efficiency; the block size used in the present invention is set according to the resolution of the shooting scene; different block sizes achieve different effects, and at the same time, compared with the traditional background differencing method, the effect of automatic threshold positioning can be achieved.

[0051] Specific method:

[0052] S21 Use the initialized background model to perform differencing on the current input picture sequence to obtain a difference image;

[0053] S22 Divide the differential image into blocks, calculate the mean and variance of each block respectively, and roughly divide the foreground and the foreground and background through a threshold.

[0054] S23 Perform a secondary rough block division on the foreground and background to obtain a block image.

[0055] S3 Perform a fine segmentation on the block image, eliminate the false foreground, and update the background.

[0056] In the embodiment of the present invention, since the background image may have certain illumination changes or certain deviations over time, the method misdetects the background as the foreground, which is called the false foreground. It is necessary to eliminate the false foreground by calculating the covariance of each pixel point in the foreground block and comparing it with the threshold; when there is a foreground in the fourth step, use the deep learning framework to perform inference on the image, calculate the intersection over union of the inference result obtained and the result of the traditional method, and when it reaches the threshold, it can be recognized as a true foreground. The combination of deep learning and the traditional method not only avoids the disadvantage of relatively poor robustness of the traditional method, but also better makes up for the low accuracy of the deep learning method; the image after eliminating the false foreground is the image containing the true foreground. In order for the method to adapt to gradual changes and sudden illumination changes, it is necessary to update the method model according to certain rules, update the weight factor according to the variance, and store the image in an array with a fixed length for the next model update, which greatly improves the robustness of the method.

[0057] Specifically, as follows Figure 4 For the results after the secondary rough segmentation and the fine segmentation results, after the secondary rough segmentation is completed, more refined segmentation needs to be carried out according to certain rules. Due to the characteristics of the rough block based on its own mean and variance for rough positioning and fixed-point photographing, in order to avoid detecting false foregrounds when the foreground target does not exist or there is a sudden illumination change and eliminate the deviation caused by the fixed-point photographing method, further fine segmentation is required, which can not only accurately segment the foreground target, but also improve the accuracy and robustness of the method;

[0058] By traversing each block obtained by the rough segmentation, calculate the covariance matrix of the average relative brightness change matrix of each foreground pixel region according to the following formulas (6) and (7), and perform segmentation according to the set threshold to obtain an accurate fine segmentation result. At the same time, through the calculation of the covariance, it can well avoid the regional misdetection caused by sudden illumination changes and has good generalization ability for complex and changeable industrial production environments;

[0059] Average relative brightness change matrix:

[0060] Foreground pixel covariance:

[0061] Among them, I b (x, y) respectively represent the foreground pixel and the corresponding pixel of the background template. In the formula, N is the specified size of the divided region. In the present invention, the size of the divided region after the second rough segmentation is directly used, and the pixels at the edge are filled and replaced with the central pixel I f (x, y) of the divided region center; different covariance thresholds are set according to different detection scenarios. If it exceeds this threshold, it can be considered that there are significant differences in this region of the current image, and the possibility of being a foreground target is the greatest;

[0062] Common object detection models such as SSD, ResNet, YOLO and other network models are used for training. The present invention conducts transfer learning based on Yolov8, so that a detection model with high accuracy can be obtained with a small amount of data, alleviating the problems of difficult acquisition and annotation of industrial data sets; as follows Figure 5 , when other interferences such as other objects and strong light changes appear in the background to be detected, traditional methods will have false detections. At this time, deep learning needs to be combined for detection. Due to the high robustness of deep learning targets, according to formula (8), the false detection area can be well masked, and at the same time, the detection of foreground targets can be ensured.

[0063] Calculation of the intersection over union (IoU) of the detection boxes:

[0064] Among them, Area(X∩T) represents the area of the intersection of detection boxes X and T, and MinArea(X, T) represents the minimum area value of detection boxes X and T; when the IoU is greater than the threshold, the Area(X∩T) region is used as the final detection box Y as the final result.

[0065] The present invention uses the method of taking fixed-interval fixed-point photos to establish a model. A model established only in the early stage is not applicable to long-term monitoring and detection. Therefore, a light weight factor update rule, such as formula (9), is used. At the same time, according to the idea of the C language queue, a fixed-length Vector container is used to store the photo sequence to achieve dynamic update, further improving the light robustness and generalization ability of the method. In the present invention, the light weight factor is obtained by weighting the mean μ b of the current background model and the mean μ t (x, y) of the current image T t ;

[0066]

[0067] Among them, N is the length of the image queue, M(x, y) is the updated background model. In the present invention, according to the photographing interval time, a storage length of 10 images is used, and β is the update rate constant, and the reference range is between [0.04, 3].

[0068] The above-disclosed is only a preferred embodiment of the foreground detection method for discrete image sequences based on background difference of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A foreground detection method for discrete image sequences based on background difference, characterized in that: The following steps are involved: Take pictures of the pipeline and sample to obtain a sequence of pictures, build a background model and initialize it; Using the initialized background model, the image sequence is roughly divided into front and back background blocks and secondary rough blocks to obtain a block image; The block images are finely segmented to eliminate pseudo foregrounds and update backgrounds.

2. The method for detecting foreground of discrete image sequences based on background difference as claimed in claim 1, It is characterized by: The specific method of the pipeline taking pictures and sampling to obtain a picture sequence, building a background model and initializing it is as follows: Build an industrial camera photography structure platform, and select a fixed time interval for photography sampling to obtain a picture sequence; A background model is established through discrete images, and the background model is initialized and subsequently updated.

3. The method for detecting foreground of discrete image sequences based on background difference as claimed in claim 1, It is characterized by: The specific method of using the initialized background model to perform coarse front and back background block division and secondary coarse block division on the picture sequence to obtain the block pictures is as follows: Using the initialized background model to differentiate the currently input picture sequence to obtain a differential image; Divide the difference image into blocks, calculate the mean and variance of each block respectively, and roughly divide the foreground and the foreground and background by threshold value; The foreground and background are roughly divided into blocks for the second time to obtain a block picture.

4. The method for detecting foreground of discrete image sequences based on background difference as claimed in claim 1, characterized in that: The method of finely segmenting the block image, eliminating false foregrounds, and updating the background also includes, if a foreground exists, using a deep learning framework to infer the block image, and calculating the rectangular frame intersection and union ratio of the inference result and the result of the traditional method together. When the calculation result reaches a threshold, it is determined to be a true foreground.

5. The method for detecting foreground of discrete image sequences based on background difference as claimed in claim 1, characterized in that ； The updating background updates the weight factor according to the variance, and stores the picture into an array of fixed length.

Images