Object motion detection method and related device

Abstract

The invention discloses an object motion detection method and a related device, and relates to the field of object detection, and the method comprises the steps: carrying out the illumination removal of a target object image based on the illumination intensity of the target object image, and obtaining a plurality of frames of illumination-removed images; based on the previous frame of de-illumination image, the current frame of de-illumination image and the next frame of de-illumination image, obtaining a first image mask by adopting an inter-frame difference method; determining the learning rate of a Gaussian mixture distribution model based on the illumination intensity, and obtaining the Gaussian mixture distribution model based on the learning rate and the pixel value of the current frame of illumination-removed image; obtaining a second image mask based on the next frame of illumination-removed image and the Gaussian mixture distribution model; adding the first image mask and the second image mask to obtain a motion area mask image of the current frame; and detecting the motion state of the target object based on the motion area mask image. According to the invention, the influence of illumination intensity change on object motion detection can be reduced, and the object motion detection accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of object detection, and more particularly to an object motion detection method and related apparatus. Background Technology

[0002] Currently, methods for detecting moving objects mainly include inter-frame differencing and background differencing. For inter-frame differencing and background differencing, drastic changes in lighting can cause the algorithm to treat more background information as moving objects, resulting in low accuracy in motion detection. For example, during landing gear retraction and extension, the camera's captured image undergoes significant changes in lighting intensity as the doors open and close. Sudden brightening or dimming of the light can cause the algorithm to mistakenly identify pixel value changes caused by changes in light intensity as changes caused by landing gear movement, thus leading to the algorithm's failure to identify the landing gear. Summary of the Invention

[0003] In view of the above problems, this application provides a method and related apparatus for object motion detection, which can reduce the impact of changes in light intensity on object motion detection, thereby improving the accuracy of object motion detection. The specific solution is as follows:

[0004] The first aspect of this application provides a method for detecting object motion, including:

[0005] Obtain a set of target object images; wherein the set of target object images includes multiple frames of target object images;

[0006] The illumination intensity of the target object image is determined, and the target object image is subjected to de-illumination processing based on the illumination intensity to obtain multiple frames of de-illumination images;

[0007] Based on the previous frame de-illuminated image, the current frame de-illuminated image, and the next frame de-illuminated image, the first image mask is obtained using the inter-frame difference method.

[0008] The learning rate of the Gaussian mixture model is determined based on the illumination intensity, and the Gaussian mixture model is obtained based on the learning rate and the pixel values ​​of the current frame deilluminated image.

[0009] Based on the next frame of the deilluminated image and the Gaussian mixture model, a second image mask is obtained;

[0010] The first image mask and the second image mask are added together to obtain the motion region mask image of the current frame;

[0011] The motion state of the target object is detected based on the motion region mask image.

[0012] In one possible implementation, determining the learning rate of the Gaussian mixture model based on the illumination intensity includes:

[0013] When the previous frame de-illuminated image is the first frame de-illuminated image, the difference between twice the preset illumination intensity and the illumination intensity of the target object image in the previous frame is calculated to obtain the first difference. The product of the first difference and the initial weight is divided by the preset illumination intensity to obtain the learning rate of the Gaussian mixture model.

[0014] When the previous frame de-illuminated image is not the first frame de-illuminated image, the difference between twice the illumination intensity of the target object image in the previous frame and the illumination intensity of the target object image in the current frame is calculated to obtain a second difference. The product of the second difference and the previous learning rate is divided by the illumination intensity of the target object image in the previous frame to obtain the learning rate of the Gaussian mixture model. The previous learning rate is the learning rate used by the Gaussian mixture model based on the pixel values ​​of the previous frame de-illuminated image.

[0015] In one possible implementation, obtaining the Gaussian mixture model based on the learning rate and the pixel values ​​of the current frame's deilluminated image includes:

[0016] When the pixels of the current frame deilluminated image are foreground, the pixel values ​​of the current frame deilluminated image are used as the mean of the Gaussian distribution of the pixels of the current frame deilluminated image to obtain the Gaussian mixture distribution model.

[0017] When the pixels of the current frame deilluminated image are the background, the mean and covariance of the Gaussian distribution of the pixels of the current frame deilluminated image are calculated based on the learning rate and the pixel values ​​of the current frame deilluminated image to obtain the Gaussian mixture distribution model.

[0018] In one possible implementation, obtaining the second image mask based on the next frame's deilluminated image and the Gaussian mixture model includes:

[0019] The absolute value of the difference between the pixel value of the next frame of the deilluminated image and the mean of the Gaussian mixture model is compared with the covariance by a preset multiple.

[0020] If the absolute value is less than the covariance of the preset multiple, then the pixels of the next frame of the deilluminated image are determined to be the background.

[0021] If the absolute value is not less than the covariance of the preset multiple, then the pixels of the next frame of the deilluminated image are determined to be foreground pixels;

[0022] The values ​​of pixels belonging to the foreground are set to 1, and the values ​​of pixels belonging to the background are set to 0, thus obtaining the second image mask.

[0023] In one possible implementation, determining the illumination intensity of the target object image includes:

[0024] The target object image is converted to grayscale to obtain a grayscale image;

[0025] The grayscale image is input into a deep learning network to obtain a weight matrix; wherein, the deep learning network is trained based on an image with a preset illumination intensity, and the weights in the weight matrix reflect the degree of influence of the pixel on the illumination intensity;

[0026] The illumination intensity of the target object image is obtained by performing a weighted average operation on the pixel value matrix of each channel of the target object image and the weight matrix.

[0027] In one possible implementation, the target object is a landing gear;

[0028] Detecting the motion state of a target object based on the motion region mask image includes:

[0029] Edge detection is performed on the mask image of the motion area to obtain the contour information of the landing gear;

[0030] Based on the contour information of the landing gear, connected component detection is performed on the mask image of the motion area to obtain the mask image of the landing gear area;

[0031] The landing gear center axis is obtained by fitting the mask image of the landing gear area to the landing gear center axis.

[0032] The landing gear angle is determined based on the central axis of the landing gear.

[0033] A second aspect of this application provides an object motion detection system, comprising:

[0034] An image acquisition module is used to acquire a set of target object images; wherein, the set of target object images includes multiple frames of target object images;

[0035] The illumination removal processing module is used to determine the illumination intensity of the target object image and perform illumination removal processing on the target object image based on the illumination intensity to obtain multiple frames of illumination removal images.

[0036] The inter-frame difference processing module is used to obtain the first image mask based on the previous frame de-illuminated image, the current frame de-illuminated image, and the next frame de-illuminated image using the inter-frame difference method.

[0037] A Gaussian distribution model building module is used to determine the learning rate of the Gaussian mixture model based on the illumination intensity, and to obtain the Gaussian mixture model based on the learning rate and the pixel values ​​of the current frame deilluminated image.

[0038] The background difference processing module is used to obtain a second image mask based on the next frame de-illuminated image and the Gaussian mixture distribution model;

[0039] The mask image generation module is used to add the first image mask and the second image mask to obtain the motion region mask image of the current frame;

[0040] The motion detection module is used to detect the motion state of the target object based on the motion region mask image.

[0041] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the object motion detection method of the first aspect or any implementation thereof.

[0042] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the computer program so that the electronic device can implement the object motion detection method of the first aspect or any implementation thereof.

[0045] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the object motion detection method described in the first aspect or any implementation thereof.

[0046] By employing the above technical solutions, the object motion detection method and related apparatus provided in this application can remove the influence of illumination on the image by determining the illumination intensity of the target object image and performing de-illumination processing on the target object image based on the illumination intensity. Furthermore, the learning rate of the Gaussian mixture model is determined based on the illumination intensity, and the Gaussian mixture model is obtained based on the learning rate and the pixel values ​​of the de-illumination image of the current frame. The learning rate can be dynamically updated according to the actual illumination intensity, effectively reducing the impact of complex environmental changes. The first image mask obtained by the inter-frame difference method is added to the second image mask obtained based on the de-illumination image of the next frame and the Gaussian mixture model to obtain the motion state detection of the target object on the motion region mask image of the current frame. This can reduce the influence of illumination intensity changes on object motion detection, thereby improving the accuracy of object motion detection. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0048] Figure 1 A flowchart of an object motion detection method provided in this application;

[0049] Figure 2 A schematic diagram of a landing gear provided in this application;

[0050] Figure 3 An image captured by an industrial camera provided in this application;

[0051] Figure 4 A schematic diagram of a landing gear angle recognition process provided in this application;

[0052] Figure 5 This application provides a schematic diagram illustrating the detection effect of moving objects before the influence of illumination.

[0053] Figure 6 This application provides a schematic diagram of the motion object detection effect after removing the influence of illumination;

[0054] Figure 7 A schematic diagram of center axis fitting provided for this application;

[0055] Figure 8 This application provides a schematic diagram of the structure of an object motion detection system.

[0056] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0057] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0058] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0059] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0060] This application provides a method for detecting object motion. The following is a detailed description of the object motion detection method according to the accompanying drawings.

[0061] Reference Figure 1 , Figure 1 This is a flowchart of an object motion detection method provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiments of this application, an object motion detection method may include:

[0062] Step 101: Obtain a set of target object images; wherein the set of target object images includes multiple frames of target object images.

[0063] The target object image can be a landing gear image. The target object image set includes multiple frames of target object images arranged in chronological order. The time corresponding to the previous frame of the target object image is earlier than the time corresponding to the current frame of the target object image, and the time corresponding to the next frame of the target object image is later than the time corresponding to the current frame of the target object image.

[0064] Optionally, after obtaining the target object image set, multiple frames of target object images in the target object image set can be converted to grayscale to obtain grayscale images.

[0065] Step 102: Determine the illumination intensity of the target object image, and perform de-illumination processing on the target object image based on the illumination intensity to obtain multiple frames of de-illumination images.

[0066] During the capture of target object images, although the camera's automatic white balance is beneficial, changes in light intensity still cause significant differences in the captured images. Sudden brightening or darkening of light can cause algorithms such as inter-frame differencing and background differencing to mistakenly identify pixel value changes caused by light intensity variations as changes caused by object motion, thus leading to the failure of the algorithm to identify moving targets. Therefore, this application employs a deep learning-based method to estimate the illumination intensity of the target object image, and then performs de-illumination processing on the image based on the estimated illumination intensity to remove the influence of illumination changes.

[0067] In one possible implementation, determining the illumination intensity of the target object image includes:

[0068] The target object image is converted to grayscale to obtain a grayscale image;

[0069] A grayscale image is input into a deep learning network to obtain a weight matrix. The deep learning network is trained based on an image with a preset illumination intensity, and the weights in the weight matrix reflect the degree of influence of each pixel on the illumination intensity.

[0070] The illumination intensity of the target object image is obtained by performing a weighted average operation on the pixel value matrix of each channel of the target object image and the weight matrix.

[0071] The target object image is a color image. Converting it to grayscale is necessary; in practical applications, the grayscale image can be converted to a 256*256 pixel size to fit the input of a deep learning network. The grayscale image is then input into the deep learning network, which can have a U-shaped network structure, resulting in a 256*256 pixel weight matrix as the output. This deep learning network is trained on an image with a preset illumination intensity, and the weights in the weight matrix reflect the degree of influence of each pixel on the illumination intensity. A weighted average of the pixel value matrix of each channel of the target object image with the weight matrix is ​​then performed. This averages the pixel value matrix of each channel of the original RGB color image with the obtained weight matrix to obtain the illumination intensity of the target object image, i.e., the estimated illumination intensity I. a =(I r ,I g ,I b ), I r I is the estimated value of the illumination intensity of the R channel. g I is the estimated value of the illumination intensity of channel G. b This is the estimated illumination intensity for channel B. r The calculation formula is as follows:

[0072]

[0073] In the formula, H is the height of the target object image, W is the width of the target object image, and X... pq Let ω be the pixel value in the p-th row and q-th column of the pixel value matrix of the target object image. pq Let be the weight in the p-th row and q-th column of the weight matrix. g and I b The calculation formula and I r Similarly, I will not go into details here.

[0074] After obtaining the estimated light intensity, a white balance algorithm is used to perform white balance processing on the image to initially remove the influence of light.

[0075] In practical applications, removing illumination from a target object image based on light intensity can be achieved by performing white balance processing on the target object image based on light intensity, resulting in multiple frames of removed illumination images. This application uses a white balance algorithm based on light intensity estimation, which can remove the influence of illumination on the image.

[0076] Step 103: Based on the previous frame deilluminated image, the current frame deilluminated image, and the next frame deilluminated image, the first image mask is obtained using the inter-frame difference method.

[0077] In one possible implementation, a first image mask is obtained using the inter-frame difference method based on the previous frame's deilluminated image, the current frame's deilluminated image, and the next frame's deilluminated image, including:

[0078] Calculate the absolute value of the difference between the previous frame's deilluminated image and the current frame's deilluminated image to obtain the first difference image;

[0079] Calculate the absolute value of the difference between the current frame's deilluminated image and the next frame's deilluminated image to obtain the second difference image;

[0080] The first difference image is binarized to obtain the first binarized image; the second difference image is binarized to obtain the second binarized image.

[0081] The intersection of the first binarized image and the second binarized image is calculated to obtain the first image mask.

[0082] When calculating the intersection of two binarized images, the intersection value is 0 if two corresponding pixel values ​​are both 0, 1 if two corresponding pixel values ​​are both 1, and 0 if two corresponding pixel values ​​are both 0 and 1. A pixel value of 1 represents the foreground, and a pixel value of 0 represents the background.

[0083] Step 104: Determine the learning rate of the Gaussian mixture model based on the illumination intensity, and obtain the Gaussian mixture model based on the learning rate and the pixel values ​​of the current frame's deilluminated image.

[0084] In one possible implementation, the learning rate of the Gaussian mixture model is determined based on the illumination intensity, including:

[0085] When the previous frame of the deilluminated image is the first frame of the deilluminated image, calculate the difference between twice the preset illumination intensity and the illumination intensity of the target object image in the previous frame to obtain the first difference. Divide the product of the first difference and the initial weight by the preset illumination intensity to obtain the learning rate of the Gaussian mixture model.

[0086] If the previous frame of the deilluminated image is not the first frame of the deilluminated image, calculate the difference between twice the illumination intensity of the target object image in the previous frame and the illumination intensity of the target object image in the current frame to obtain the second difference. Divide the product of the second difference and the previous learning rate by the illumination intensity of the target object image in the previous frame to obtain the learning rate of the Gaussian mixture model. The previous learning rate is the learning rate used by the Gaussian mixture model based on the pixel values ​​of the previous frame of the deilluminated image.

[0087] The update of the Gaussian mixture model is related to the learning rate. To improve the adaptability of the Gaussian mixture model to changes in illumination, this application dynamically updates the learning rate based on the illumination intensity. When determining the learning rate of the Gaussian mixture model based on the illumination intensity, it is distinguished whether the previous frame of the deilluminated image is the first frame of the deilluminated image. If the previous frame of the deilluminated image is the first frame of the deilluminated image, the learning rate is calculated based on the illumination intensity of the target object image in the previous frame, the preset illumination intensity, and the initial weights. Here, the initial weights are set values ​​and are constants. The preset illumination intensity can be the same as the preset illumination intensity in the deep learning network trained based on the image with the preset illumination intensity in step 102, and can both be standard illumination intensity. If the previous frame of the deilluminated image is not the first frame of the deilluminated image, that is, the previous frame of the deilluminated image is the 2nd, 3rd, ... Nth frame of the deilluminated image, the learning rate is calculated based on the illumination intensity of the target object image in the previous frame, the illumination intensity of the target object image in the current frame, and the pixel value of the previous frame of the deilluminated image to obtain the learning rate used by the Gaussian mixture model. The formula for calculating the learning rate of a Gaussian mixture model is as follows:

[0088]

[0089] In the formula, a is the frame number of the previous frame of the deilluminated image, ω' is the learning rate of the Gaussian mixture model, and ω a The learning rate used by the Gaussian mixture model is determined based on the pixel values ​​of the previous frame's unlit image. This represents the illumination intensity of the target object image in the previous frame, specifically the average illumination intensity of the three channels of the target object image in the previous frame. ω0 is the illumination intensity of the target object image in the current frame, which is the average of the illumination intensity of the three channels of the target object image in the current frame. ω0 is the initial weight, ω is a set value constant, and I0 is the preset illumination intensity.

[0090] In one possible implementation, a Gaussian mixture model is obtained based on the learning rate and the pixel values ​​of the current frame's unilluminated image, including:

[0091] When the pixels of the current frame's deilluminated image are foreground, the pixel values ​​of the current frame's deilluminated image are used as the mean of the Gaussian distribution of the pixels in the current frame's deilluminated image, thus obtaining a Gaussian mixture distribution model.

[0092] When the pixels of the current frame's deilluminated image are the background, the mean and covariance of the Gaussian distribution of the pixels in the current frame's deilluminated image are calculated based on the learning rate and the pixel values ​​of the current frame's deilluminated image, resulting in a Gaussian mixture distribution model.

[0093] To determine whether a pixel in the current frame's deilluminated image is foreground or background, the absolute value of the difference between the pixel value and the mean of the Gaussian mixture model can be compared to a preset multiple of the covariance. If the absolute value is less than the preset multiple of the covariance, it is background; otherwise, it is foreground. This preset multiple can be 2.5.

[0094] Based on the fundamental principles of the Gaussian mixture model, we first establish k Gaussian distributions for each pixel in the first frame of the lit-down image, where k can be 3. The probability distribution of the pixels is as follows:

[0095]

[0096] In the formula, p(x) is the probability distribution of the pixel, and ω s,t Let η(x) be the weight of the s-th Gaussian distribution at time t, and let the first frame of the deilluminated image be the deilluminated image at time t. t ,μ s,t ,δ s,t Let x be the s-th Gaussian distribution at time t. t Let μ be the pixel value of the pixel at time t. s,t Let δ be the mean of the s-th Gaussian distribution at time t. s,t Let be the covariance of the s-th Gaussian distribution at time t.

[0097] Based on the aforementioned Gaussian distribution, the pixels of the second frame of the de-illuminated image are matched pixel by pixel. By comparing each pixel sequentially with the established k Gaussian distributions, x is determined. t+1 Is it foreground or background? t+1 Let x be the pixel value of the pixel at time t+1. t Let x be the pixel value of the first frame of the unlit image. t+1For the second frame of the image, remove the illumination from the pixels. The applied formula is: If |(x t+1 -μ s,t )|<2.5δ s,t Then the second frame's unlit image pixels are the background, if |(x t+1 -μ s,t )|≥2.5δ s,t Then the pixels in the second frame of the deilluminated image are the foreground.

[0098] When the pixels in the current frame's deilluminated image are foreground, the pixel values ​​of the current frame's deilluminated image are used as the mean of the Gaussian distribution of the pixels in the current frame's deilluminated image, resulting in a Gaussian mixture distribution model. If the current frame is the second frame, then when the pixels in the second frame's deilluminated image are foreground, a new Gaussian distribution is created using the pixel values ​​of the second frame's deilluminated image as the mean of the Gaussian distribution of the pixels in the second frame's deilluminated image. This new distribution replaces the k Gaussian distributions established based on the first frame's deilluminated image, where the distribution with the lowest weight (ω) is the one with the lowest weight. s,t The minimum Gaussian distribution is obtained, thus yielding the mixture Gaussian distribution model.

[0099] When the pixels of the current frame's deilluminated image are the background, the mean and covariance of the Gaussian distribution of the pixels in the current frame's deilluminated image are calculated based on the learning rate and the pixel values ​​of the current frame's deilluminated image, resulting in a Gaussian mixture model. If the current frame is the second frame, then when the pixels of the second frame's deilluminated image are the background, the mean and covariance of the Gaussian distribution of the pixels in the current frame's deilluminated image are calculated according to the following formula:

[0100] μ i =(1-α)μ i-1 +αx i

[0101]

[0102] in,

[0103]

[0104] In the formula, μ i Let μ be the mean of the Gaussian distribution of the pixels in the current frame's unilluminated image i. i-1 x is the mean of the Gaussian distribution of pixels i-1 in the previous frame's unlit image. i The pixel values ​​of the current frame's unlit image, α is an intermediate parameter, and δ i Let δ be the covariance of the Gaussian distribution of pixels in the current frame's unlit image i. i-1 Let ω' be the covariance of the Gaussian distribution of pixels i-1 in the previous frame's unlit image, and ω' be the learning rate.

[0105] Step 105: Based on the next frame's deilluminated image and the Gaussian mixture model, obtain the second image mask.

[0106] In one possible implementation, a second image mask is obtained based on the next frame's deilluminated image and a Gaussian mixture model, including:

[0107] The absolute value of the difference between the pixel value of the next frame of the deilluminated image and the mean of the Gaussian mixture model is compared with the covariance by a preset multiple.

[0108] If the absolute value is less than a preset multiple of the covariance, then the pixels of the next frame of the deilluminated image are determined to be the background.

[0109] If the absolute value is not less than the covariance of a preset multiple, then the pixels of the next frame of the deilluminated image are determined as the foreground.

[0110] Set the values ​​of pixels belonging to the foreground to 1 and the values ​​of pixels belonging to the background to 0 to obtain the second image mask.

[0111] When comparing the absolute value of the difference between the pixel values ​​of the next frame of the deilluminated image and the mean of the Gaussian mixture model with the covariance of a preset multiple, the Gaussian mixture model is η(x i ,μ i ,δ i The preset multiple is 2.5. If it conforms to the formula: |(x i+1 -μ i )|<2.5δ i Then determine the next frame's deilluminated image x. i+1 If the pixel value is the background, then if it does not conform to the formula, i.e., |(x i+1 -μ i )|≥2.5δ i Then determine the next frame's deilluminated image x. i+1 The pixels belonging to the foreground are set to 1, and the pixels belonging to the background are set to 0, thus obtaining the second image mask.

[0112] Step 106: Add the first image mask and the second image mask together to obtain the motion region mask image of the current frame.

[0113] When a pixel in the second frame of the de-illuminated image is the background, it can be set to 0; when a pixel is the foreground, it can be set to 1. This yields an image mask based on background subtraction corresponding to the second frame of the de-illuminated image. The image mask based on inter-frame subtraction corresponding to the second frame of the de-illuminated image can be obtained using inter-frame subtraction based on the first, second, and third frames of the de-illuminated image. Specifically, the absolute value of the difference between the first and second frames of the de-illuminated image is calculated and binarized. The absolute value of the difference between the second and third frames of the de-illuminated image is also calculated and binarized. The intersection of the two binarized images yields the image mask based on inter-frame subtraction corresponding to the second frame of the de-illuminated image. Finally, the motion region mask image for the second frame is obtained by combining the image mask based on inter-frame subtraction corresponding to the second frame of the de-illuminated image and the image mask based on inter-frame subtraction corresponding to the second frame of the de-illuminated image.

[0114] Step 107: Detect the motion state of the target object based on the motion region mask image.

[0115] In one possible implementation, the target object is a landing gear, and the motion state detection of the target object is performed based on a motion region mask image, including:

[0116] Edge detection is performed on the mask image of the moving area to obtain the contour information of the landing gear;

[0117] Based on the contour information of the landing gear, connected component detection is performed on the mask image of the motion area to obtain the mask image of the landing gear area.

[0118] The landing gear centerline is obtained by fitting the mask image of the landing gear area to the landing gear centerline.

[0119] The landing gear angle is determined based on the landing gear centerline.

[0120] Optionally, before performing edge detection on the motion region mask image, a denoising operation can be performed on the motion region mask image. In practical applications, the mask image can be dilated and then eroded to remove noise data generated during the calculation process.

[0121] When performing connected component detection on the motion region mask image based on the landing gear contour information, since the landing gear retraction and extension may be accompanied by the retraction and extension of the hatch, the hatch part will also be retained when performing moving object detection. Therefore, the region with a quadrilateral connected component and an aspect ratio of less than 1:10 can be set as the background to eliminate the interference of the landing gear hatch and obtain the mask image of the landing gear region.

[0122] When fitting the landing gear centerline to the mask image of the landing gear area, the Hough line detection algorithm can be used to fit the landing gear centerline and obtain the vector coordinates of the centerline. When determining the landing gear angle based on the landing gear centerline, the angle between the centerline and the horizontal vector can be calculated based on the vector coordinates of the centerline, thereby obtaining the landing gear angle.

[0123] By reading the images frame by frame in sequence, the above steps can detect moving objects and their angles of motion.

[0124] This application employs a non-contact recognition method. Through illumination removal processing, motion object detection based on three-part frame difference and background difference, and angle detection, it can achieve landing gear angle state detection under complex environmental changes. By determining the illumination intensity of the target object image and performing illumination removal processing based on the illumination intensity, the influence of illumination on the image can be removed. Furthermore, the learning rate of the Gaussian mixture model is determined based on the illumination intensity, and the Gaussian mixture model is obtained based on the learning rate and the pixel values ​​of the current frame's illumination-removed image. The learning rate can be dynamically updated according to the actual illumination intensity, effectively reducing the impact of complex environmental changes. The first image mask obtained by the inter-frame difference method is added to the second image mask obtained based on the next frame's illumination-removed image and the Gaussian mixture model. The motion region mask image of the current frame is then used to detect the target object's motion state, which can reduce the influence of illumination intensity changes on object motion detection, thereby improving the accuracy of object motion detection.

[0125] The object motion detection method provided in this application differs from traditional sensor data-based methods. It does not rely on the analysis and calculation of real-time landing gear parameters collected through aircraft system sensor data, nor does it rely on the traditional method of capturing landing gear images using binocular cameras and modeling the landing gear using 3D reconstruction technology to infer the real-time operating status of the landing gear. Instead, it uses a monocular camera and motion detection image processing method to identify the landing gear retraction angle.

[0126] To further illustrate the object motion detection method provided in this application, a landing gear can be used as an example of the target object. The target object image is a landing gear image; a schematic diagram of the landing gear image can be found in [reference needed]. Figure 2 A high frame rate industrial camera is installed inside the landing gear bay to capture and record images of the bay in real time. Images captured by the industrial camera can be used as a reference. Figure 3 . Figure 2 In this process, industrial cameras transmit real-time captured images to a recognition algorithm terminal via a network. The recognition algorithm terminal performs image processing to remove the effects of illumination, locates the landing gear area, and performs image recognition to obtain the landing gear extension and retraction angle. At the same time, the recognition results and real-time images are fed back to the display terminal through the avionics network.

[0127] A schematic diagram of the landing gear angle recognition process is shown below. Figure 4 As shown, video data captured by an industrial camera undergoes illumination removal processing. During this process, the image's illumination intensity is estimated, and white balance processing is performed based on the estimation results to initially remove illumination effects. Then, the illumination-removed image is sequentially processed using the three-frame difference method and background subtraction method. This background subtraction method is an improved Gaussian mixture model (GaM) background subtraction method. First, background initialization is performed, i.e., initializing the Gaussian mixture model to obtain k Gaussian distributions. Then, subtraction operations are performed, i.e., pixel matching is performed on the next frame image based on the initialized Gaussian model to determine whether it is foreground or background. The Gaussian mixture model is then updated. During model update, the learning rate of the Gaussian mixture distribution model is determined based on the illumination intensity, and the updated model is obtained based on the learning rate to perform background subtraction operations on subsequent image frames. The image mask obtained by the background subtraction method and the image mask obtained by the three-frame difference method are ANDed together to obtain the motion region mask image. Then, landing gear angle recognition is performed. First, noise data can be removed by dilation and denoising operations. Then, edge detection is performed to obtain the contour information of the landing gear. Then, connected component detection is performed to obtain the mask image of the landing gear area. Finally, angle calculation is performed, that is, the landing gear center axis is fitted to the mask image of the landing gear area to obtain the landing gear center axis. The real-time landing gear angle is obtained based on the landing gear center axis.

[0128] Figure 5 This is a schematic diagram illustrating the detection effect of moving objects before removing the influence of lighting. Figure 6 This is a schematic diagram illustrating the motion object detection effect after removing the influence of lighting. Figure 5 and Figure 6 The comparison shows that the method provided in this application is more effective at reducing the impact of complex environmental changes. Figure 7 This is a schematic diagram of the center axis fitting, where the red line represents the fitted landing gear center axis.

[0129] The above describes an object motion detection method provided by an embodiment of this application. The following describes a system for performing the above object motion detection method.

[0130] The object motion detection system provided in this application embodiment, such as Figure 8 As shown, the system includes:

[0131] The image acquisition module 801 is used to acquire a set of target object images; wherein the set of target object images includes multiple frames of target object images.

[0132] The illumination removal processing module 802 is used to determine the illumination intensity of the target object image and perform illumination removal processing on the target object image based on the illumination intensity to obtain multiple frames of illumination removal images.

[0133] The inter-frame difference processing module 803 is used to obtain the first image mask based on the previous frame de-illuminated image, the current frame de-illuminated image, and the next frame de-illuminated image using the inter-frame difference method.

[0134] The Gaussian distribution model establishment module 804 is used to determine the learning rate of the Gaussian mixture model based on the illumination intensity, and to obtain the Gaussian mixture model based on the learning rate and the pixel values ​​of the current frame's deilluminated image.

[0135] Background difference processing module 805 is used to obtain a second image mask based on the next frame's deilluminated image and a Gaussian mixture distribution model.

[0136] The mask image generation module 806 is used to add the first image mask and the second image mask to obtain the motion region mask image of the current frame.

[0137] The motion detection module 807 is used to detect the motion state of a target object based on a motion region mask image.

[0138] In one possible implementation, the Gaussian distribution model building module 804 is specifically used for:

[0139] When the previous frame of the deilluminated image is the first frame of the deilluminated image, calculate the difference between twice the preset illumination intensity and the illumination intensity of the target object image in the previous frame to obtain the first difference. Divide the product of the first difference and the initial weight by the preset illumination intensity to obtain the learning rate of the Gaussian mixture model.

[0140] If the previous frame of the deilluminated image is not the first frame of the deilluminated image, calculate the difference between twice the illumination intensity of the target object image in the previous frame and the illumination intensity of the target object image in the current frame to obtain the second difference. Divide the product of the second difference and the previous learning rate by the illumination intensity of the target object image in the previous frame to obtain the learning rate of the Gaussian mixture model. The previous learning rate is the learning rate used by the Gaussian mixture model based on the pixel values ​​of the previous frame of the deilluminated image.

[0141] In one possible implementation, the Gaussian distribution model building module 804 is also used for:

[0142] When the pixels of the current frame's deilluminated image are foreground, the pixel values ​​of the current frame's deilluminated image are used as the mean of the Gaussian distribution of the pixels in the current frame's deilluminated image, thus obtaining a Gaussian mixture distribution model.

[0143] When the pixels of the current frame's deilluminated image are the background, the mean and covariance of the Gaussian distribution of the pixels in the current frame's deilluminated image are calculated based on the learning rate and the pixel values ​​of the current frame's deilluminated image, resulting in a Gaussian mixture distribution model.

[0144] In one possible implementation, the background difference processing module 805 is specifically used for:

[0145] The absolute value of the difference between the pixel value of the next frame of the deilluminated image and the mean of the Gaussian mixture model is compared with the covariance by a preset multiple.

[0146] If the absolute value is less than a preset multiple of the covariance, then the pixels of the next frame of the deilluminated image are determined to be the background.

[0147] If the absolute value is not less than the covariance of a preset multiple, then the pixels of the next frame of the deilluminated image are determined as the foreground.

[0148] Set the values ​​of pixels belonging to the foreground to 1 and the values ​​of pixels belonging to the background to 0 to obtain the second image mask.

[0149] In one possible implementation, the illumination removal processing module 802 is specifically used for:

[0150] The target object image is converted to grayscale to obtain a grayscale image;

[0151] A grayscale image is input into a deep learning network to obtain a weight matrix. The deep learning network is trained based on an image with a preset illumination intensity, and the weights in the weight matrix reflect the degree of influence of each pixel on the illumination intensity.

[0152] The illumination intensity of the target object image is obtained by performing a weighted average operation on the pixel value matrix of each channel of the target object image and the weight matrix.

[0153] In one possible implementation, the target object is the landing gear. The motion detection module 807 is specifically used for:

[0154] Target object motion state detection based on motion region mask images includes:

[0155] Edge detection is performed on the mask image of the moving area to obtain the contour information of the landing gear;

[0156] Based on the contour information of the landing gear, connected component detection is performed on the mask image of the motion area to obtain the mask image of the landing gear area.

[0157] The landing gear centerline is obtained by fitting the mask image of the landing gear area to the landing gear centerline.

[0158] The landing gear angle is determined based on the landing gear centerline.

[0159] This application also provides an electronic device in its embodiments. (See reference...) Figure 9The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0160] like Figure 9 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. When the electronic device is powered on, the RAM 903 also stores various programs and data required for the operation of the electronic device. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0161] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, memory cards, hard drives, etc.; and communication devices 909. Communication device 909 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0162] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the object motion detection methods provided in this application.

[0163] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the object motion detection methods provided in this application.

[0164] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0165] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0166] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0167] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for detecting object motion, characterized in that, include: Obtain a set of target object images; wherein the set of target object images includes multiple frames of target object images; The illumination intensity of the target object image is determined, and the target object image is subjected to de-illumination processing based on the illumination intensity to obtain multiple frames of de-illumination images; Based on the previous frame de-illuminated image, the current frame de-illuminated image, and the next frame de-illuminated image, the first image mask is obtained using the inter-frame difference method. The learning rate of the Gaussian mixture model is determined based on the illumination intensity, and the Gaussian mixture model is obtained based on the learning rate and the pixel values ​​of the current frame deilluminated image. Based on the next frame of the deilluminated image and the Gaussian mixture model, a second image mask is obtained; The first image mask and the second image mask are added together to obtain the motion region mask image of the current frame; The motion state of the target object is detected based on the motion region mask image.

2. The object motion detection method according to claim 1, characterized in that, The determination of the learning rate for the Gaussian mixture model based on the illumination intensity includes: When the previous frame de-illuminated image is the first frame de-illuminated image, the difference between twice the preset illumination intensity and the illumination intensity of the target object image in the previous frame is calculated to obtain the first difference. The product of the first difference and the initial weight is divided by the preset illumination intensity to obtain the learning rate of the Gaussian mixture model. When the previous frame de-illuminated image is not the first frame de-illuminated image, the difference between twice the illumination intensity of the target object image in the previous frame and the illumination intensity of the target object image in the current frame is calculated to obtain a second difference. The product of the second difference and the previous learning rate is divided by the illumination intensity of the target object image in the previous frame to obtain the learning rate of the Gaussian mixture model. The previous learning rate is the learning rate used by the Gaussian mixture model based on the pixel values ​​of the previous frame de-illuminated image.

3. The object motion detection method according to claim 1, characterized in that, The method for obtaining the Gaussian mixture model based on the learning rate and the pixel values ​​of the current frame's deilluminated image includes: When the pixels of the current frame deilluminated image are foreground, the pixel values ​​of the current frame deilluminated image are used as the mean of the Gaussian distribution of the pixels of the current frame deilluminated image to obtain the Gaussian mixture distribution model. When the pixels of the current frame deilluminated image are the background, the mean and covariance of the Gaussian distribution of the pixels of the current frame deilluminated image are calculated based on the learning rate and the pixel values ​​of the current frame deilluminated image to obtain the Gaussian mixture distribution model.

4. The object motion detection method according to claim 1, characterized in that, The process of obtaining a second image mask based on the next frame's de-illuminated image and the Gaussian mixture model includes: The absolute value of the difference between the pixel value of the next frame of the deilluminated image and the mean of the Gaussian mixture model is compared with the covariance by a preset multiple. If the absolute value is less than the covariance of the preset multiple, then the pixels of the next frame of the deilluminated image are determined to be the background. If the absolute value is not less than the covariance of the preset multiple, then the pixels of the next frame of the deilluminated image are determined to be foreground pixels; The values ​​of pixels belonging to the foreground are set to 1, and the values ​​of pixels belonging to the background are set to 0, thus obtaining the second image mask.

5. The object motion detection method according to any one of claims 1 to 4, characterized in that, Determining the illumination intensity of the target object image includes: The target object image is converted to grayscale to obtain a grayscale image; The grayscale image is input into a deep learning network to obtain a weight matrix; wherein, the deep learning network is trained based on an image with a preset illumination intensity, and the weights in the weight matrix reflect the degree of influence of the pixel on the illumination intensity; The illumination intensity of the target object image is obtained by performing a weighted average operation on the pixel value matrix of each channel of the target object image and the weight matrix.

6. The object motion detection method according to any one of claims 1 to 4, characterized in that, The target object is a landing gear; Detecting the motion state of a target object based on the motion region mask image includes: Edge detection is performed on the mask image of the motion area to obtain the contour information of the landing gear; Based on the contour information of the landing gear, connected component detection is performed on the mask image of the motion area to obtain the mask image of the landing gear area; The landing gear center axis is obtained by fitting the mask image of the landing gear area to the landing gear center axis. The landing gear angle is determined based on the central axis of the landing gear.

7. An object motion detection system, characterized in that, include: An image acquisition module is used to acquire a set of target object images; wherein, the set of target object images includes multiple frames of target object images; The illumination removal processing module is used to determine the illumination intensity of the target object image and perform illumination removal processing on the target object image based on the illumination intensity to obtain multiple frames of illumination removal images. The inter-frame difference processing module is used to obtain the first image mask based on the previous frame de-illuminated image, the current frame de-illuminated image, and the next frame de-illuminated image using the inter-frame difference method. A Gaussian distribution model building module is used to determine the learning rate of the Gaussian mixture model based on the illumination intensity, and to obtain the Gaussian mixture model based on the learning rate and the pixel values ​​of the current frame deilluminated image. The background difference processing module is used to obtain a second image mask based on the next frame de-illuminated image and the Gaussian mixture distribution model; The mask image generation module is used to add the first image mask and the second image mask to obtain the motion region mask image of the current frame; The motion detection module is used to detect the motion state of the target object based on the motion region mask image.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the object motion detection method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the object motion detection method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the object motion detection method as described in any one of claims 1 to 6.

Images