Moving object detection method and device based on optical flow network
Through the method based on optical flow network, the optical flow information field and motion characteristics in the video stream are extracted, and the misjudgment and fuzzy recognition problems of moving objects detection in the prior art are solved, thereby achieving higher detection accuracy and accuracy. In the detection of complex scenes and high-speed objects.
Patent Information
- Application Number
- CN202410936896.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
The existing methods of detecting moving objects have high misjudgment rates when facing light changes, background dynamic changes and fast moving objects, making it difficult to accurately identify objects that have stayed for a long time, and have poor recognition of fuzzy objects, affecting the detection effect.
Using an optical flow network-based method, by acquiring a video stream, extracting an optical flow information field, obtaining the motion characteristics of each pixel in the image, and extracting moving objects based on these characteristics. The specific steps include inputting the video stream into the optical flow network, extracting feature images using downsampled convolutional layer and sliding window converter, calculating matching probability and small-scale optical flow field, performing optical flow propagation and self-attention calculation, and finally based on the velocity threshold. Determine the area of moving objects.
Improve the detection accuracy of moving objects, can handle lighting changes and background dynamic changes more accurately, effectively capture motion patterns in complex scenes, and is suitable for object detection of different speeds and sizes, especially in security and on-board camera applications. Higher detection accuracy.
Smart Images

Figure CN118918149A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to a moving object detection method and device based on an optical flow network. Background Art
[0002] Moving object detection plays a vital role in many practical applications, such as security monitoring, vehicle-mounted cameras, traffic flow monitoring, etc. Traditional moving object detection methods are usually based on background modeling or frame difference. For example, the background modeling method detects the foreground or moving objects by building a background model and comparing it with the real-time frame; the frame difference method determines the moving object by comparing the differences between consecutive frames.
[0003] However, the existing traditional methods are not ideal in practical applications. The main reasons are as follows:
[0004] 1. Changes in illumination, dynamic background changes (such as wind blowing leaves, raindrops, etc.) or shadows may lead to misjudgment of moving objects, thus affecting the accuracy of detection results;
[0005] 2. When an object stays for a long time, it is likely to be mistaken for the background, so when it moves, the algorithm cannot identify it;
[0006] 3. Fast-moving objects may cause blurring of objects in the captured images, which may make the algorithm unable to recognize the objects due to insufficient clarity, thus affecting the detection effect.
[0007] To address these problems, researchers have tried to introduce more complex background modeling methods or combine multiple algorithms to enhance robustness, but the results are still limited. Summary of the invention
[0008] The purpose of the present invention is to provide a moving object detection method and device based on optical flow network to solve the problem of how to effectively detect moving objects.
[0009] In order to solve the above technical problems, the present invention provides a moving object detection method based on an optical flow network, comprising:
[0010] Acquire a video stream, wherein the video stream includes at least 2 consecutive frames of images;
[0011] Using the optical flow network, the optical flow information field is extracted from continuous images;
[0012] Through the optical flow information field, the motion characteristics of each pixel in the image are obtained;
[0013] Extract moving objects based on motion features.
[0014] Optionally, in the moving object detection method based on the optical flow network, the method of extracting the optical flow information field from the continuous images using the optical flow network includes:
[0015] Input the video stream into the optical flow network;
[0016] Use the downsampling convolutional layer in the optical flow network to extract feature images;
[0017] The sliding window converter in the optical flow network is used to enhance the feature image.
[0018] The feature matrix is obtained according to the feature images after two frames of continuous feature enhancement;
[0019] The flexible maximum transfer function in the optical flow network is used to compare the feature similarity of all pixels in the feature image to obtain the matching probability;
[0020] Use matching probability to perform global feature matching to obtain a small-scale optical flow field;
[0021] The upsampling convolution layer in the optical flow network is used to upsample the small-scale optical flow field to guide the feature image to perform optical flow propagation and obtain an optical flow feature map;
[0022] The self-attention is calculated for the optical flow feature map, and the optical flow is propagated again, and the optical flow information field is obtained after upsampling.
[0023] Optionally, in the moving object detection method based on the optical flow network, the method of extracting the feature image using the downsampling convolution layer in the optical flow network includes:
[0024] The downsampling convolution layer in the optical flow network is used to downsample two adjacent frame images to extract the first feature image and the second feature image.
[0025] Optionally, in the moving object detection method based on the optical flow network, the method of using the sliding window converter in the optical flow network to perform feature enhancement on the feature image includes:
[0026] Set a learnable fixed position encoding to confirm the position relationship between pixels;
[0027] The second feature image and the fixed position code are input into the sliding window converter together to enhance the features of the second feature image.
[0028] Optionally, in the moving object detection method based on optical flow network, the sliding window converter is composed of multiple sliding window converters superimposed in series.
[0029] Optionally, in the moving object detection method based on the optical flow network, the calculation formula for performing global feature matching using matching probability to obtain a small-scale optical flow field is:
[0030] Flow mini =S×GG
[0031] G=interpo(F 1 ,F 2 )
[0032] S = softmax(Z)
[0033]
[0034] in, In Indicates the feature dimension, H, W, and D represent the height, width, and number of channels of the feature dimension respectively; Flow mini represents the small-scale optical flow field, G represents the grid of the optical flow field corresponding to the optical flow output, interpo(,) represents bilinear interpolation; S represents the matching probability of pixel feature similarity obtained by the feature matrix Z through the flexible maximum transfer function; Z represents the feature matrix, F 1 and F 2 They are the feature images after feature enhancement by sliding window converter,
[0035] Optionally, in the moving object detection method based on the optical flow network, the method of upsampling the small-scale optical flow field by using the upsampling convolution layer in the optical flow network to guide the feature image to perform optical flow propagation to obtain the optical flow feature map includes:
[0036] Upsample the small-scale optical flow field to obtain the optical flow features;
[0037] Set the previous frame image as the reference frame image;
[0038] Using the optical flow feature, performing optical flow correction on the first feature image corresponding to the non-reference frame image;
[0039] Optical flow propagation is performed on the first feature image after optical flow correction corresponding to the non-reference frame image to obtain an optical flow feature map.
[0040] Optionally, in the moving object detection method based on the optical flow network, the method of calculating self-attention on the optical flow feature map, performing optical flow propagation again, and obtaining the optical flow information field after upsampling includes:
[0041] Performing self-attention calculation on the first feature image and the optical flow feature map corresponding to the reference frame image to obtain an attention feature map;
[0042] The attention feature map is upsampled and convolved in sequence to obtain the optical flow information field.
[0043] Optionally, in the moving object detection method based on the optical flow network, the method of obtaining the motion feature of each pixel in the image through the optical flow information field includes:
[0044] According to the optical flow information field, calculate the speed of each pixel in the image;
[0045] Set speed threshold;
[0046] If the speed of the pixel is greater than the speed threshold, the optical flow vector corresponding to the pixel is retained, and the optical flow vector is a motion feature.
[0047] Optionally, in the moving object detection method based on the optical flow network, the method of extracting the moving object according to the motion features includes:
[0048] From the pixel points corresponding to the retained optical flow vector, find the maximum and minimum values of the horizontal and vertical coordinates;
[0049] According to the maximum and minimum values of the horizontal and vertical coordinates, the area range of the moving object in the image is confirmed;
[0050] Based on the region range, a bounding box is formed to mark the moving object.
[0051] In order to solve the above technical problems, the present invention further provides a moving object detection device based on an optical flow network, which is used to implement the moving object detection method based on an optical flow network as described in any one of the above items, and the moving object detection device based on an optical flow network includes:
[0052] A video input module, used to obtain a video stream, wherein the video stream includes at least 2 frames of continuous images;
[0053] An optical flow network module, comprising a pyramid optical flow deformation network; the pyramid optical flow deformation network comprises a downsampling convolution layer, a sliding window converter, a flexible maximum transfer function, an upsampling convolution layer and a self-attention unit; the optical flow network module is used to obtain the motion characteristics of pixels from a video stream using the pyramid optical flow deformation network;
[0054] The object detection module is used to extract moving objects based on motion features.
[0055] The present invention provides a moving object detection method and device based on an optical flow network, comprising: obtaining a video stream, the video stream comprising at least 2 frames of continuous images; using an optical flow network to extract an optical flow information field from the continuous images; obtaining the motion features of each pixel in the image through the optical flow information field; and extracting moving objects based on the motion features. By extracting the motion features of pixels through an optical flow network, it is possible to more accurately process changes in illumination, dynamic changes in the background, and shadows, thereby being able to more accurately capture and understand motion patterns in complex scenes, so that moving objects of different speeds and sizes can be effectively detected, solving the problem of how to effectively detect moving objects. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 A flow chart of a moving object detection method based on an optical flow network provided in this embodiment;
[0057] Figure 2 A schematic diagram of the process of the moving object detection method based on optical flow network provided in this embodiment;
[0058] Figure 3 This is a structural block diagram of the moving object detection device based on optical flow network provided in this embodiment. DETAILED DESCRIPTION
[0059] The following is a further detailed description of the moving object detection method and device based on optical flow network proposed in the present invention in conjunction with the accompanying drawings and specific embodiments. It should be noted that the accompanying drawings are all in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In addition, the structure shown in the accompanying drawings is often a part of the actual structure. In particular, the emphasis that each drawing needs to show is different, and sometimes different proportions are used.
[0060] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects in order to describe the embodiments of the present invention, rather than to describe a specific order or sequence. It should be understood that the structures used in this way can be interchanged under appropriate circumstances. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0061] Optical flow network is a technology used for image processing, mainly used to estimate the motion vectors of pixels or small blocks in video sequences over time, which constitute the optical flow field. The basic concepts involved in the operation of optical flow network include: optical flow, which refers to the motion trajectory of points in space over time, and describes the curve of each pixel position changing over time. It consists of translation field (displacement field) and phase field. The translation field describes the amount of movement of the pixel position over time, while the phase field is related to factors such as lighting and camera perspective.
[0062] The structure of the commonly used optical flow network includes a contraction part and an enlargement part. The contraction part is mainly composed of convolutional layers, which are used to extract the features of the two images; the enlargement part can intelligently restore the optical flow to high pixels to provide dense optical flow prediction.
[0063] Based on this, this embodiment provides a moving object detection method based on optical flow network, such as Figure 1 As shown, including:
[0064] S1, obtaining a video stream, wherein the video stream includes at least 2 frames of continuous images;
[0065] S2, using the optical flow network, extracts the optical flow information field from the continuous images;
[0066] S3, obtains the motion characteristics of each pixel in the image through the optical flow information field;
[0067] S4, extracting moving objects according to the motion features.
[0068] The moving object detection method based on optical flow network provided in this embodiment can more accurately process lighting changes, background dynamic changes and shadows by extracting the motion features of pixels through the optical flow network, so as to more accurately capture and understand the motion patterns in complex scenes, so that moving objects of different speeds and sizes can be effectively detected, solving the problem of how to effectively detect moving objects.
[0069] Specifically, in this embodiment, step S1, a video stream is obtained, wherein the video stream includes at least 2 consecutive frames of images. In practical applications, the video stream may be 2 consecutive frames of images or video frames, so that the optical flow network can be used to analyze and process the 2 frames of images to obtain the motion vector (motion feature) of the pixel in time.
[0070] It should be noted that the video stream described in this embodiment includes at least 2 frames of continuous images, which may be multiple frames of continuous images (such as a video stream including image 1, image 2, ... image n, where image 1, image 2, ... image n are continuous in time frame), or a combination of at least 2 frames of continuous images and several other frames of discontinuous images (such as a video stream including image 1, image 3, image 4, ... image n, where image 3 and image 4 are 2 frames of images continuous in time frame, and image 1 and image 3 are discontinuous); as long as the video stream contains 2 frames of continuous images, it is considered to include at least 2 frames of continuous images. This application does not limit the total number of frames of images contained in the video stream, the arrangement of continuous frame images, etc.
[0071] Furthermore, in this embodiment, if Figure 2 As shown, step S2, using an optical flow network, the method for extracting an optical flow information field from a continuous image includes:
[0072] S21, input the video stream into the optical flow network.
[0073] Specifically, in this embodiment, two consecutive frames of images are input, wherein the previous (earlier in time) frame is defined as a reference frame I1, and the other frame is defined as a non-reference frame I2.
[0074] And, the optical flow network provided in this embodiment is specifically a pyramid optical flow deformation network; the pyramid optical flow deformation network includes a downsampling convolution layer Dconv, a sliding window converter, a flexible maximum transfer function softmax, an upsampling convolution layer Uconv, a self-attention unit, and an output convolution layer conv_out, etc. The functions and effects of each structure in the pyramid optical flow deformation network will be described in detail below, and its architecture is well known to those skilled in the art, and this application will not repeat it.
[0075] S22, extract feature images using the downsampling convolutional layer in the optical flow network.
[0076] Specifically, in this embodiment, the downsampling convolution layer includes a downsampling convolution layer 1 (Dconv1) and a downsampling convolution layer 2 (Dconv2), so as to downsample the two frames of images respectively. Among them, the downsampling convolution layer 1 can downsample the reference frame I1 and the non-reference frame I2 respectively to obtain a first feature image corresponding to the reference frame I1 and the non-reference frame I2, wherein the size of the first feature image is 1 / 2 of the original image size; the downsampling convolution layer 2 can downsample the first feature image output by the downsampling convolution layer 1 again to obtain a second feature image corresponding to the reference frame I1 and the non-reference frame I2, wherein the size of the second feature image is 1 / 4 of the original image size.
[0077] In this embodiment, the downsampling formula of the downsampling convolution layer 1 (Dconv1) can be expressed as:
[0078] F 1_1 ,F 2_1 = Down_conv 1 (I 1 ,I 2 )
[0079] And, the downsampling formula of the downsampling convolution layer 2 (Dconv2) can be expressed as:
[0080] F 1_2 ,F 2_2 = Down_conv 2 (F 1_1 ,F 2_1 )
[0081] At this time, the downsampling convolution layer obtains the first feature image F of 1 / 2 size corresponding to the reference frame I1 1_1 and the second feature image F of 1 / 4 size 1_2 , and the first feature image F of 1 / 2 size corresponding to the non-reference frame I2 2_1 and the second feature image F of 1 / 4 size 2_2 .
[0082] S23, uses the sliding window converter in the optical flow network to enhance the feature image.
[0083] Specifically, in this embodiment, the sliding window converter is composed of a plurality of sliding window converters connected in series and superimposed on each other, thereby improving the effect of feature enhancement.
[0084] In addition, in this embodiment, in order to correspond to the pixel position relationship between the two frames of images, a learnable fixed position code, denoted as P, can be set first to confirm the position relationship between the pixels; then the second feature image and the fixed position code are input into the sliding window converter together to perform feature enhancement on the second feature image.
[0085] In this embodiment, the formula for performing feature enhancement on the second feature image using the sliding window converter can be expressed as:
[0086] F 1 ,F 2 =SwinT ×n (F 1_2 +P,F 2_2 +P)
[0087] Among them, SwinT ×n It represents n sliding window converters superimposed in series.
[0088] After the second feature image is enhanced by using the sliding window converter, the enhanced second feature image corresponding to the reference frame I1 is F 1 , the enhanced second feature image corresponding to the non-reference frame I2 is F 2 .
[0089] S24, obtaining a feature matrix according to two frames of feature-enhanced feature images.
[0090] Specifically, in this embodiment, the feature images after feature enhancement of two consecutive frames are dimensionally transposed and then matrix multiplied to obtain a feature matrix. That is, the enhanced second feature image F corresponding to the reference frame I1 is multiplied by 1 The enhanced second feature image F corresponding to the non-reference frame I2 2 After transposing the dimensions, matrix multiplication is performed. The calculation formula can be expressed as:
[0091]
[0092] Among them, Z represents the feature matrix, In It represents the feature dimension, and H, W, and D represent the height, width, and number of channels of the feature dimension respectively.
[0093] S25, using the flexible maximum transfer function in the optical flow network to compare the feature similarities of all pixels in the feature image to obtain a matching probability.
[0094] Specifically, in this embodiment, the calculation formula of the matching probability can be expressed as:
[0095] S = softmax(Z)
[0096] Among them, S represents the matching probability of pixel feature similarity obtained by the feature matrix Z through the soft maximum transfer function softmax.
[0097] S26, using the matching probability to perform global feature matching to obtain a small-scale optical flow field.
[0098] Specifically, in this embodiment, the enhanced second feature image F corresponding to the reference frame I1 is first 1 The enhanced second feature image F corresponding to the non-reference frame I2 2 Bilinear interpolation is performed to obtain the corresponding optical flow output grid. At this time, the number of output channels D of the optical flow field is 2; then, the coordinates of the optical flow output grid are weighted averaged by matching probability to obtain a small-scale optical flow field of 1 / 4 size.
[0099] In this embodiment, the calculation formula of the grid output by the optical flow can be expressed as:
[0100] G=interpo(F 1 ,F 2 )
[0101] And, the calculation formula of the small-scale optical flow field can be expressed as:
[0102] Flow mini =S×GG
[0103] Among them, G represents the grid of the optical flow field corresponding to the optical flow output, In this embodiment, G represents the grid corresponding to the optical flow output when the number of optical flow field output channels is 2. interpo(,) means bilinear interpolation; Flow mini represents the small-scale optical flow field, In this embodiment, corresponding to G,
[0104] S27, uses the upsampling convolution layer in the optical flow network to upsample the small-scale optical flow field to guide the feature image to perform optical flow propagation and obtain an optical flow feature map.
[0105] Specifically, in this embodiment, the upsampling convolution layer includes an upsampling convolution layer 1 (Uconv1) and an upsampling convolution layer 2 (Uconv2).
[0106] First, the small-scale optical flow field is upsampled using the upsampling convolution layer 1 (Uconv1) to obtain a 1 / 2-sized optical flow feature, thereby guiding the 1 / 2-sized first feature image to perform optical flow propagation. The calculation formula of the 1 / 2-sized optical flow feature can be expressed as:
[0107] F up =Up_conv 1 (Flow mini )
[0108] Among them, F up To use the upsampling convolution layer 1 to the small-scale optical flow field Flow mini The 1 / 2 size optical flow features are obtained after upsampling.
[0109] Then, the first feature image F corresponding to the non-reference frame image I2 is obtained by using the optical flow feature. 2_1 Perform optical flow correction, where the calculation formula of the optical flow feature obtained by the optical flow correction can be expressed as:
[0110] F warp =Warp(F 2_1 ,F up )
[0111] Among them, Fwarp is the optical flow feature obtained by optical flow correction, that is, the optical flow feature map; Warp(,) represents the optical flow correction operation, using the optical flow feature F up The first feature image F corresponding to the non-reference frame image I2 2_1 Perform optical flow correction.
[0112] By the first feature image F 2_1 Optical flow correction is performed so that when optical flow propagation is performed on the first feature image corresponding to the non-reference frame image I2 after optical flow correction, the optical flow feature map obtained has higher accuracy.
[0113] S28, calculate the self-attention of the optical flow feature map, and propagate the optical flow again, and obtain the optical flow information field after upsampling.
[0114] Specifically, in this embodiment, first, the first feature image F of 1 / 2 size corresponding to the reference frame image I1 is 1_1 And the optical flow feature map F warp Perform self-attention calculation to obtain the attention feature map, where the formula for self-attention calculation can be expressed as:
[0115] M=Attn(F warp ,F 1_1 )
[0116] Among them, M is the attention feature map, and Attn(,) represents the self-attention calculation.
[0117] Then, the attention feature map is upsampled and convolved in turn to obtain the optical flow information field. Among them, the upsampling is implemented by the upsampling convolution layer 2 (Uconv2), and the convolution is implemented by the output convolution layer conv_out. The calculation formula of the optical flow information field can be expressed as:
[0118] Flow=Conv out (Up_conv 2 (M))
[0119] Among them, Flow represents the optical flow field image, Up_conv 2 Indicates upsampling using upsampling convolution layer 2, Conv out Indicates that convolution processing is performed using the output convolution layer.
[0120] After obtaining the optical flow information field (optical flow field image), in this embodiment, step S3, a method for obtaining the motion feature of each pixel in the image through the optical flow information field includes:
[0121] S32, calculating the speed of each pixel in the image according to the optical flow information field.
[0122] Specifically, in this embodiment, the calculation formula of the pixel speed can be expressed as:
[0123]
[0124] Among them, V represents the speed of the pixel; T represents the time interval between two frames of images, which can be obtained by frame rate calculation; (μ, ν) represents the displacement vector corresponding to the pixel in the optical flow field. By calculating the Euclidean distance of the pixel's displacement vector, the pixel's speed of movement can be obtained.
[0125] S33, setting a speed threshold.
[0126] Specifically, the speed threshold is recorded as C. In practical applications, the speed threshold needs to be reasonably set according to actual needs. Only when the speed V of the pixel is greater than the speed threshold C, the pixel is identified as a moving pixel, thereby avoiding misjudgment of moving objects due to displacement vectors of pixels caused by scene lighting changes, background dynamic changes, shadows, etc.
[0127] S34: If the speed of the pixel is greater than the speed threshold, retain the optical flow vector corresponding to the pixel, where the optical flow vector is a motion feature.
[0128] By setting a speed threshold and comparing the speed of the pixel with the speed threshold, it can be ensured that the extracted motion pixels correspond to the moving object, thereby improving the accuracy of detecting the moving object.
[0129] Furthermore, in this embodiment, in step S4, the method of extracting the moving object according to the motion feature includes:
[0130] S41, finding the maximum and minimum values of the horizontal and vertical coordinates from the pixel points corresponding to the retained optical flow vectors.
[0131] Specifically, in this embodiment, all extracted pixel points are traversed to find the maximum and minimum values of the horizontal coordinates, which are recorded as Xmax and Xmin respectively, and the maximum and minimum values of the vertical coordinates, which are recorded as Ymax and Ymin respectively.
[0132] S42, confirming the area range of the moving object in the image according to the maximum and minimum values of the horizontal and vertical coordinates.
[0133] Specifically, the maximum and minimum values of the horizontal and vertical coordinates are arranged and combined to obtain four pixel coordinates, namely (Xmax, Ymax), (Xmax, Ymin), (Xmin, Ymax) and (Xmin, Ymin), which correspond to the lower right corner, upper right corner, lower left corner and upper left corner of the moving object area in the image, respectively, so as to frame a rectangular area circumscribed to the moving object, that is, the minimum area covering the moving object.
[0134] S43: Based on the region range, a bounding box is formed to mark the moving object.
[0135] Connecting the four coordinate points in sequence can form a bounding box. Since the bounding box is circumscribed to the moving object, the moving object can be clearly and accurately marked and extracted.
[0136] The moving object detection method based on the optical flow network provided in this embodiment, thanks to the optical flow network, can use the movement between image pixels to estimate the movement direction and speed of objects in the scene. At the same time, by adding a deep learning neural network to the optical flow network, it can more accurately capture and understand the motion patterns in complex scenes.
[0137] The implementation results show that compared with the existing moving object detection methods, the moving object detection method based on the optical flow network provided in this embodiment can more accurately handle the interference of lighting changes, background dynamic changes and shadows, thereby improving the detection accuracy of moving objects; the moving object detection method based on the optical flow network provided in this embodiment has a good detection effect on moving objects of different speeds and sizes, especially in security and vehicle-mounted camera applications, it has a higher detection accuracy for the detection of pedestrians and vehicles; the moving object detection method based on the optical flow network provided in this embodiment has a higher detection accuracy rate whether in crowded urban streets or in complex outdoor environments, especially for high-speed moving objects and objects in low-light environments.
[0138] This embodiment also provides a moving object detection device based on an optical flow network, which is used to implement the moving object detection method based on an optical flow network as described above. Figure 3 As shown, the moving object detection device based on optical flow network includes:
[0139] A video input module, used to obtain a video stream, wherein the video stream includes at least 2 frames of continuous images;
[0140] An optical flow network module, comprising a pyramid optical flow deformation network; the pyramid optical flow deformation network comprises a downsampling convolution layer, a sliding window converter, a soft maximum transfer function softmax, an upsampling convolution layer and a self-attention unit; the optical flow network module is used to obtain the motion characteristics of pixels from a video stream using the pyramid optical flow deformation network;
[0141] The object detection module is used to extract moving objects based on motion features.
[0142] The moving object detection device based on optical flow network provided in this embodiment can more accurately process lighting changes, background dynamic changes and shadows by extracting the motion features of pixels through the optical flow network module, so as to more accurately capture and understand the motion patterns in complex scenes, so that moving objects of different speeds and sizes can be effectively detected, solving the problem of how to effectively detect moving objects.
[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other. In addition, the different parts between the various embodiments can also be used in combination with each other, and the present invention is not limited to this.
[0144] The present embodiment provides a moving object detection method and device based on an optical flow network, including: obtaining a video stream, the video stream including at least 2 frames of continuous images; using an optical flow network to extract an optical flow information field from the continuous images; obtaining the motion features of each pixel in the image through the optical flow information field; and extracting moving objects based on the motion features. By extracting the motion features of pixels through an optical flow network, it is possible to more accurately process changes in illumination, dynamic changes in the background, and shadows, thereby being able to more accurately capture and understand motion patterns in complex scenes, so that moving objects of different speeds and sizes can be effectively detected, solving the problem of how to effectively detect moving objects.
[0145] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.
Claims
1. A moving object detection method based on optical flow network, It is characterized in that include: Acquire a video stream, wherein the video stream includes at least 2 consecutive frames of images; Using the optical flow network, the optical flow information field is extracted from continuous images; Through the optical flow information field, the motion characteristics of each pixel in the image are obtained; Extract moving objects based on motion features.
2. The moving object detection method based on optical flow network according to claim 1, It is characterized in that The method of extracting the optical flow information field from the continuous images by using the optical flow network comprises: Input the video stream into the optical flow network; Use the downsampling convolutional layer in the optical flow network to extract feature images; The sliding window converter in the optical flow network is used to enhance the feature image. The feature matrix is obtained according to the feature images after two frames of continuous feature enhancement; The flexible maximum transfer function in the optical flow network is used to compare the feature similarity of all pixels in the feature image to obtain the matching probability; Use matching probability to perform global feature matching to obtain a small-scale optical flow field; The upsampling convolution layer in the optical flow network is used to upsample the small-scale optical flow field to guide the feature image to perform optical flow propagation and obtain an optical flow feature map; The self-attention is calculated for the optical flow feature map, and the optical flow is propagated again, and the optical flow information field is obtained after upsampling.
3. The moving object detection method based on optical flow network according to claim 2, It is characterized in that The method for extracting a feature image using a downsampling convolutional layer in an optical flow network comprises: The downsampling convolution layer in the optical flow network is used to downsample two adjacent frame images to extract the first feature image and the second feature image.
4. The moving object detection method based on optical flow network according to claim 3, It is characterized in that The method for enhancing the features of a feature image by using a sliding window converter in an optical flow network comprises: Set a learnable fixed position encoding to confirm the position relationship between pixels; The second feature image and the fixed position code are input into the sliding window converter together to perform feature enhancement on the second feature image.
5. The moving object detection method based on optical flow network according to claim 2, It is characterized in that The sliding window converter is composed of a plurality of sliding window converters which are connected in series and superimposed on each other.
6. The moving object detection method based on optical flow network according to claim 2, It is characterized in that The calculation formula for obtaining the small-scale optical flow field by performing global feature matching using the matching probability is: Flow mini =S×G-G G=interpolate(F 1 ,F 2 ) S = softmax(Z) in, In Indicates the feature dimension, H, W, and D represent the height, width, and number of channels of the feature dimension respectively; Flow mini represents the small-scale optical flow field, G represents the grid of the optical flow field corresponding to the optical flow output, interpo(,) represents bilinear interpolation; S represents the matching probability of pixel feature similarity obtained by the feature matrix Z through the flexible maximum transfer function; Z represents the feature matrix, F 1 and F 2 They are the feature images after feature enhancement by sliding window converter, 7. The moving object detection method based on optical flow network as claimed in claim 3, It is characterized in that The method of upsampling a small-scale optical flow field by using an upsampling convolution layer in an optical flow network to guide the feature image to perform optical flow propagation and obtain an optical flow feature map includes: Upsample the small-scale optical flow field to obtain the optical flow features; Set the previous frame image as the reference frame image; Using the optical flow feature, performing optical flow correction on the first feature image corresponding to the non-reference frame image; Optical flow propagation is performed on the first feature image after optical flow correction corresponding to the non-reference frame image to obtain an optical flow feature map.
8. The moving object detection method based on optical flow network as claimed in claim 7, It is characterized in that The method of calculating self-attention on the optical flow feature map, and then performing optical flow propagation again to obtain the optical flow information field after upsampling includes: Performing self-attention calculation on the first feature image and the optical flow feature map corresponding to the reference frame image to obtain an attention feature map; The attention feature map is upsampled and convolved in sequence to obtain the optical flow information field.
9. The moving object detection method based on optical flow network according to claim 1, It is characterized in that The method for obtaining the motion feature of each pixel in the image through the optical flow information field includes: According to the optical flow information field, calculate the speed of each pixel in the image; Set speed threshold; If the speed of the pixel is greater than the speed threshold, the optical flow vector corresponding to the pixel is retained, and the optical flow vector is a motion feature.
10. The moving object detection method based on optical flow network according to claim 9, It is characterized in that The method for extracting a moving object according to the motion characteristics comprises: From the pixel points corresponding to the retained optical flow vector, find the maximum and minimum values of the horizontal and vertical coordinates; According to the maximum and minimum values of the horizontal and vertical coordinates, the area range of the moving object in the image is confirmed; Based on the region range, a bounding box is formed to mark the moving object.
11. A moving object detection device based on an optical flow network, used to implement the moving object detection method based on an optical flow network as claimed in any one of claims 1 to 10, It is characterized in that The moving object detection device based on optical flow network includes: A video input module, used to obtain a video stream, wherein the video stream includes at least 2 frames of continuous images; An optical flow network module, comprising a pyramid optical flow deformation network; the pyramid optical flow deformation network comprises a downsampling convolution layer, a sliding window converter, a flexible maximum transfer function, an upsampling convolution layer and a self-attention unit; the optical flow network module is used to obtain the motion characteristics of pixels from a video stream using the pyramid optical flow deformation network; The object detection module is used to extract moving objects based on motion features.
