An unattended system and its monitoring method based on optical flow

Abstract

The application discloses an unattended system and a monitoring method thereof based on optical flow, which comprises the following steps: acquiring images taken by the unattended system at a preset fixed frequency, pre-processing the images, and selecting three continuous image frames according to the capture time of the image frames; inputting the three pre-processed image frames into a preset optical flow estimation model to estimate an optical flow map from a current frame to a next frame; acquiring camera internal parameters, and estimating a scene flow map according to the optical flow map and the camera internal parameters; acquiring a time difference from the current frame to the next frame according to the capture time of the image frames, and obtaining a pixel-by-pixel motion speed pixel by pixel according to the scene flow map and the time difference from the current frame to the next frame; and performing connected region division and marking according to the pixel-by-pixel motion speed and a preset speed threshold to obtain a moving target and a motion speed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, in particular relates to an unattended system and a monitoring method thereof based on optical flow. BACKGROUND

[0002] With the development of automation in various industries, the unattended system has a wide range of applications in civil and military fields. The current unattended system usually uses visible light cameras, infrared cameras and the like, and locates and identifies targets by constructing a target detection model.

[0003] The disadvantages of the prior art are: first, the existing target detection model is usually constructed by supervised deep learning, and the knowledge base established thereby enables it to only identify the species categories seen, and does not have the detection capability for categories not learned, or objects disguised, or objects obscured by smoke. Therefore, for these scenarios, the unattended system based on the target detection neural network model will fail. Second, the unattended system using only the target detection model has very limited capability, it can only obtain the target category and the position in the picture, and cannot estimate the position and motion speed of the target in the real world, and it is difficult to judge the danger level. Therefore, it is urgent to provide a more effective and complete unattended system and monitoring method. SUMMARY

[0004] In view of the above technical problems, the present application provides an unattended system and a monitoring method thereof based on optical flow, which is suitable for complex environments, has finer granularity of identification, provides more abundant information, and has higher cost performance.

[0005] The technical solution adopted by the present application to solve its technical problems is:

[0006] An unattended system based on optical flow, the method comprising the following steps:

[0007] Step S100: acquiring images taken by the unattended system at a preset fixed frequency, pre-processing the images, and selecting three consecutive image frames according to the capture time of the image frames;

[0008] Step S200: inputting the three pre-processed image frames into a preset optical flow estimation model to estimate the optical flow map from the current frame to the next frame;

[0009] Step S300: acquiring the camera intrinsic parameters, and estimating the scene flow map from the optical flow map and the camera intrinsic parameters;

[0010] Step S400: acquiring the time difference from the current frame to the next frame according to the capture time of the image frames, and obtaining the pixel-by-pixel motion speed pixel by pixel according to the scene flow map and the time difference from the current frame to the next frame;

[0011] Step S500: Based on the pixel-by-pixel motion speed and a preset speed threshold, the connected regions are divided and marked to obtain the moving target and motion speed.

[0012] Preferably, the preprocessing involves uniformly cropping the image frames to a fixed size W×H, where W represents the number of pixels in the width of the cropped image and H represents the number of pixels in the height of the cropped image.

[0013] Preferably, the preset optical flow estimation model includes an encoding module, an attention module, and an iterative update module. The encoding module includes a feature encoder and a context encoder, both of which are composed of convolutional networks. The attention module includes a temporal attention aggregator and a spatial attention aggregator. The iterative update module is composed of a GRU.

[0014] Preferably, the three consecutive image frames are the previous frame, the current frame, and the next frame, and are named Frame I. t-1 , frame I t and frame I t+1 Step S200 includes:

[0015] Step S210: Frame I t-1 , frame I t and frame I t+1 The feature tensor and context tensor for each frame are extracted by the feature encoder and context encoder, respectively.

[0016] Step S220: For frame I t-1 and frame I t The feature tensors are used to calculate correlation, and the resulting motion correlation tensors are then convolved and encoded to obtain frame I. t-1 Motion characteristics For frame I t and frame I t+1 The correlation of the feature tensor is calculated, and the resulting motion correlation tensor and the optical flow obtained from the previous iteration of the GRU module are convolved and encoded to obtain frame I. t Motion characteristics Mt raw ;

[0017] Step S230: Transfer frame I t-1 and frame I t context tensor f t-1 ,f t The input time attention aggregator calculates the mutual attention coefficient as the time attention coefficient α. temporal Based on the temporal attention coefficient, the previous frame I t-1 Motion characteristics By clustering along the time dimension, time-related features are obtained.

[0018] Step S240: copying the context tensor of frame t twice and inputting into the spatial attention aggregator to calculate the self-attention coefficient as the spatial attention coefficient spatial , according to the spatial attention coefficient , the motion feature of the current frame t is aggregated in the spatial dimension to obtain the spatial correlation feature

[0019] Step S250: inputting the context tensor, motion feature , time correlation feature , and spatial correlation feature of the current frame t into the iterative update module to update the optical flow through a series of GRU modules to obtain the optical flow map of the current frame I t to the next frame I t+1 with the original resolution of WxH.

[0020] Preferably, step S230 is specifically:

[0021]

[0022]

[0023] wherein, denotes the correlation coefficient value at the i-th row and j-th column of the time correlation matrix of the current frame I t and the previous frame I t-1 , f t；(i) is the i-th column vector of the context tensor of the current frame I t , f t-1；(j) is the j-th column vector of the context tensor of the previous frame I t-1 , D c denotes the feature dimension of the context tensor, W q denotes the first trainable query matrix, W k denotes the first trainable key matrix; denotes the i-th column vector of the obtained time correlation feature , f denotes the i-th column vector of the current frame motion feature , f denotes the normalized time correlation coefficient at (i, j), denotes the j-th column vector of the previous frame motion feature , W v denotes the first trainable value matrix, and N is the total number of columns of the matrix.

[0024] Step S240 is specifically:

[0025]

[0026]

[0027] wherein, denotes the i-th column vector of the context tensor, t denotes the correlation coefficient value at the i-th row and j-th column of the spatial correlation matrix, t；(i) denotes the i-th column vector of the context tensor, t denotes the i-th column vector of the context tensor, c denotes the feature dimension of the context tensor, q′ denotes the second trainable query matrix, k′ denotes the second trainable key matrix, denotes the i-th column vector of the resulting spatial correlation feature, denotes the i-th column vector of the resulting spatial correlation feature, denotes the i-th column vector of the current frame motion feature, denotes the i-th column vector of the current frame motion feature, denotes the normalized spatial correlation coefficient at (i, j), v′ denotes the second trainable value matrix,

[0028] Preferably, the step S400 is specifically:

[0029]

[0030] wherein, denotes the motion velocity of any pixel point p on the current frame I t , and Δt denotes the time difference from the current frame to the next frame, denotes the depth of the pixel point p of the frame I t+1 , and Δt denotes the time difference from the current frame to the next frame, denotes the depth of the pixel point p of the frame I t , and Δt denotes the time difference from the current frame to the next frame.

[0031] Preferably, the step S500 comprises:

[0032] Step S510: comparing the pixel-wise motion velocity with a preset velocity threshold value, and retaining the pixels in the velocity map whose motion velocity is greater than the preset velocity threshold value;

[0033] Step S520: segmenting the velocity according to a preset segmentation principle, constructing connected regions of different levels, regarding the connected regions whose velocity difference is within a preset range as a whole, outputting the connected regions as the motion target, and regarding the average velocity of the connected regions as the motion velocity.

[0034] An unattended system comprises a wheeled mobile robot, a standard visible light camera and a computing platform, the standard visible light camera is arranged on the wheeled mobile robot, the wheeled mobile robot advances according to a preset velocity, the standard visible light camera shoots the front, and the computing platform is used for executing a light flow-based monitoring method in the unattended system to obtain a motion target and a motion velocity.

[0035] The unmanned system and the monitoring method based on the optical flow have stronger generalization ability, higher precision, lower cost, are suitable for complex environments, have finer granularity of identification, provide richer information, and have higher cost performance. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A flowchart of the monitoring method based on the optical flow in the unmanned system in an embodiment of the present application.

[0037] Figure 2 A comparison diagram of the optical flow algorithm and the existing GMA algorithm in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0039] In one embodiment, as shown in Figure 1 The monitoring method based on the optical flow in the unmanned system comprises the following steps:

[0040] Step S100: acquiring images captured by the unmanned system at a preset fixed frequency, pre-processing the images, and selecting three continuous image frames according to the capture time of the image frames.

[0041] Specifically, the unmanned system comprises a wheeled mobile robot, a standard visible light camera, and a computing platform. The standard visible light camera is arranged on the wheeled mobile robot. The wheeled mobile robot advances according to a preset speed. The standard visible light camera captures an area in front of the wheeled mobile robot and captures images at a preset fixed frequency.

[0042] In one embodiment, the pre-processing is to uniformly crop the image frames into a fixed size WxH, where W represents the number of pixels of the width of the cropped image, and H represents the number of pixels of the height of the cropped image.

[0043] Step S200: inputting the three pre-processed image frames into a preset optical flow estimation model to estimate an optical flow map from a current frame to a next frame.

[0044] Specifically, the preset optical flow estimation model is a novel network model combining space-time attention. The motion vector of each pixel of the middle frame to the next frame constitutes the optical flow map of the middle frame.

[0045] In one embodiment, the preset optical flow estimation model comprises an encoding module, an attention module, and an iterative updating module. The encoding module comprises a feature encoder and a context encoder, both of which are composed of a convolutional network. The attention module comprises a time attention aggregator and a space attention aggregator. The iterative updating module is composed of a GRU.

[0046] In one embodiment, the three consecutive image frames are a previous frame, a current frame and a next frame, respectively named frame I t-1 , frame I t and frame I t+1 , and step S200 comprises:

[0047] Step S210: frame I t-1 , frame I t and frame I t+1 extract corresponding feature tensors and context tensors of each frame through feature encoders and context encoders respectively;

[0048] Step S220: calculate the correlation of the feature tensors of frame I t-1 and frame I t , and encode the obtained motion correlation tensor through convolution operation to obtain the motion feature t-1 of frame I t ; Step S220: calculate the correlation of the feature tensors of frame I t and frame I t+1 , and encode the obtained motion correlation tensor and the optical flow output by the GRU module in the last iteration to obtain the motion feature t of frame I t-1 ;

[0049] Step S230: input the context tensors f t-1 , f t of frame I t-1 and frame I t to the temporal attention aggregator to calculate the mutual attention coefficient as the temporal attention coefficient a temporal , and aggregate the motion feature of the previous frame I t-1 in the time dimension according to the temporal attention coefficient to obtain the time-related feature

[0050] Step S240: duplicate the context tensor of frame t twice and input it to the spatial attention aggregator to calculate the self-attention coefficient as the spatial attention coefficient a spatial , and aggregate the motion feature of the current frame t in the spatial dimension according to the spatial attention coefficient to obtain the spatial-related feature

[0051] Step S250: input the context tensor, motion feature , time-related feature and spatial-related feature of the current frame t to the iterative update module, and update the optical flow through a series of GRU modules to obtain the current frame I t to the next frame I t+1The original resolution of the optical flow map is W×H.

[0052] In one embodiment, step S230 specifically involves:

[0053]

[0054]

[0055] in, Refers to the current frame I t And the previous frame I t-1 The correlation coefficient value at the i-th row and j-th column of the time correlation matrix, f t；(i) It is the current frame I t The i-th column vector of the context tensor, f t-1；(j) It is the previous frame I t-1 The j-th column vector of the context tensor, D c W refers to the feature dimension of the context tensor. q W refers to the first trainable query matrix. k This refers to the first trainable key matrix; For the obtained time-related features The i-th column vector, Refers to the motion features of the current frame The i-th column vector, It refers to the time correlation coefficient at (i,j) after normalization. Motion features of the previous frame The j-th column vector, W v It refers to the first trainable value matrix, where N is the total number of columns in the matrix.

[0056] Specifically, formula (2) refers to normalizing the temporal correlation coefficient matrix between the current frame and the previous frame, and aggregating the motion features of the previous frame. As a timeline.

[0057] In one embodiment, step S240 specifically involves:

[0058]

[0059]

[0060] in, Refers to the current frame I t The correlation coefficient value at the i-th row and j-th column of the spatial correlation matrix, f t；(i) It is the current frame I t The i-th column vector of the context tensor, D c W refers to the feature dimension of the context tensor. q′ This refers to the second trainable query matrix, W.k′ This refers to the second trainable key matrix; For the obtained spatial correlation features The i-th column vector, Refers to the motion features of the current frame The i-th column vector, W refers to the normalized spatial correlation coefficient at (i,j). v′ It refers to the second trainable value matrix, where N is the total number of columns in the matrix.

[0061] Step S300: Obtain camera intrinsic parameters and estimate the scene flow graph based on the optical flow graph and camera intrinsic parameters.

[0062] Step S400: Obtain the time difference between the current frame and the next frame based on the capture time of the image frame, and obtain the pixel-by-pixel motion speed based on the scene flow graph and the time difference between the current frame and the next frame.

[0063] Specifically, assuming an object L occupies pixels l and l′ before and after motion, respectively, and depths Z and Z′, the pixel count can be calculated from the camera focal length f and depth, i.e., l = fL / Z, l′ = fL / Z′, establishing the relationship between optical flow scale changes and depth scale changes Z′ / Z. Assuming a point's coordinates in the image coordinate system of two consecutive frames are p and p′, and its coordinates in the 3D coordinate system are P and P′, combined with the camera intrinsic parameter matrix K, we know ZP = KP, Z′p′ = Kp′, and thus calculate the scene flow s at point p. p =P′-P. For frame I t The scene flow is calculated pixel by pixel to obtain the scene flow graph S. o Combined with the current frame I t and the next frame I t+1 The time difference Δt is used to calculate the motion velocity s pixel by pixel. p / Δt, to obtain the velocity diagram V o ;

[0064] In one embodiment, step S400 specifically involves:

[0065]

[0066] in, For the current frame I t The velocity of any pixel p on the image, where Δt is the time difference between the current frame and the next frame. For frame I t+1 The depth of pixel p, For frame I t The depth of pixel p.

[0067] Step S500: Based on the pixel-by-pixel motion speed and a preset speed threshold, the connected regions are divided and marked to obtain the moving target and motion speed.

[0068] In one embodiment, step S500 includes:

[0069] Step S510: Compare the pixel-by-pixel motion speed with a preset speed threshold, and retain pixels in the speed map whose motion speed is greater than the preset speed threshold;

[0070] Step S520: Divide the velocity into segments according to the preset segmentation principle, construct connected regions at different levels, treat connected regions with velocity differences within the preset range as a whole, output the connected regions as the motion target, and use the average velocity of the connected regions as the motion velocity.

[0071] Specifically, a motion speed threshold is set to filter out moving pixels. Pixels exceeding the speed threshold (default close to jogging speed, set to 1 m / s) are segmented into four equal parts according to speed from low to high. Connected regions are then defined and labeled. If a connected region contains fewer than 10 pixels, it is considered noise and filtered out. Thus, the method of this invention segments the moving target at the pixel level and provides the target's true motion speed.

[0072] Existing target detection algorithms in unattended systems can only identify categories already present in the training set. The technology proposed in this invention is category-independent, has stronger generalization ability, and higher accuracy, as shown in the appendix. Figure 2 The top three frames are the original three consecutive images, and the bottom two, from top to bottom, are I... t-1 I t I t+1 The optical flow maps estimated by GMA and those estimated by the algorithm involved in this invention are shown. The superiority of the method of this invention can be seen within the dashed squares. For foot movement, there is no background adhesion, meaning the background is stationary; colorless represents almost no optical flow. The algorithm involved in this invention estimates more accurately. Furthermore, it can obtain detailed target real-time motion velocity using only a standard visible light camera, without requiring expensive depth sensors such as LiDAR, effectively reducing costs.

[0073] An unattended system includes a wheeled mobile robot, a standard visible light camera, and a computing platform. The standard visible light camera is mounted on the wheeled mobile robot, which moves forward at a preset speed. The standard visible light camera captures images of the area directly in front of it. The computing platform is used to execute an optical flow-based monitoring method for the unattended system to obtain the moving target and its speed.

[0074] The above describes in detail the unattended system and the monitoring method based on the optical flow. The principle and the implementation of the present application are described by using specific examples. The above description of the examples is only used to help understand the core idea of the present application. It should be pointed out that the ordinary skilled in the art can make some improvements and modifications to the present application without departing from the principle of the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method for monitoring based on optical flow in an unattended system, characterized in that, The method comprises the following steps: Step S100: acquiring images taken by the unattended system at a preset fixed frequency, pre-processing the images, and selecting three consecutive image frames according to the capture time of the image frames; Step S200: inputting the pre-processed three image frames into a preset optical flow estimation model to estimate an optical flow map from a current frame to a next frame; the preset optical flow estimation model comprises an encoding module, an attention module, and an iterative updating module; the encoding module comprises a feature encoder and a context encoder, both of which are composed of a convolutional network; the attention module comprises a temporal attention aggregator and a spatial attention aggregator; and the iterative updating module is composed of a GRU module; Three consecutive image frames are a previous frame, a current frame and a next frame, respectively named as frame I t-1 , frame I t and frame I t+1 , step S200 comprises: Step S210: The frame I t-1 , The frame I t and The frame I t+1 are respectively extracted into the feature tensor and the context tensor corresponding to each frame by the feature encoder and the context encoder. Step S220: correlation calculation is performed on the feature tensors of the frame I t-1 and the frame I t , convolution operation encoding is performed on the obtained motion correlation tensor, and the motion feature of the frame I t-1 is obtained Correlation calculation is performed on the feature tensors of the frame I t and the frame I t+1 , convolution operation encoding is performed on the obtained motion correlation tensor and the optical flow obtained by the last iteration of the GRU module, and the motion feature of the frame I t is obtained Step S230: inputting the frame I t-1 and the context tensor f t of the frame I t-1 , f t into the time attention aggregator to calculate the mutual attention coefficients as the time attention coefficients α temporal , and performing aggregation in the time dimension on the motion features of the previous frame I t-1 according to the time attention coefficients to obtain the time-related features ​ Step S240: copying the context tensor 2 times of the frame I t and inputting the spatial attention aggregator to calculate the self-attention coefficient as the spatial attention coefficient α spatial , according to the spatial attention coefficient, the motion feature of the current frame t is aggregated in the spatial dimension to obtain the spatial correlation feature Step S250: Set the current frame I t The context tensor, the motion features The time-related features The spatial correlation features The iterative update module is input, and the optical flow is updated through a series of GRU modules to obtain the current frame I. t To the next frame I t+1 The original resolution of the optical flow map is W×H; Step S300: acquiring camera intrinsic parameters, and estimating a scene flow map from the optical flow map and the camera intrinsic parameters; Step S400: acquiring a time difference from the current frame to the next frame according to the capture time of the image frames, and obtaining a pixel-by-pixel motion speed pixel by pixel according to the scene flow map and the time difference from the current frame to the next frame; Step S500: performing connected region division and labeling according to the pixel-by-pixel motion speed and a preset speed threshold to obtain a moving target and a motion speed.

2. The method of claim 1, wherein, The pre-processing is to uniformly crop the image frames into a fixed size WxH, where W represents the number of pixels of the width of the cropped image, and H represents the number of pixels of the height of the cropped image.

3. The method of claim 2, wherein, Step S230 specifically comprises: wherein, denotes the current frame I t and the previous frame I t-1 a correlation coefficient value at the i-th row and j-th column of the temporal correlation matrix, f t ；(i) denotes the current frame I t an i-th column vector of the context tensor, f t-1；(j) denotes the previous frame I t-1 a j-th column vector of the context tensor, D c denotes a feature dimension of the context tensor, W q denotes a first trainable query matrix, W k denotes a first trainable key matrix; is an i-th column vector of the resulting temporal correlation feature denotes an i-th column vector of the current frame motion feature denotes a normalized temporal correlation coefficient at (i, j), denotes a j-th column vector of the previous frame motion feature v denotes a first trainable value matrix, N is the total number of columns of the matrix;​​​ Step S240 specifically comprises: wherein, denotes the current frame I t denotes the correlation coefficient value at the i-th row and j-th column of the spatial correlation matrix, f t；(i) denotes the current frame I t denotes the i-th column vector of the context tensor, D c denotes the feature dimension of the context tensor, W q' denotes the second trainable query matrix, W k' denotes the second trainable key matrix; denotes the resulting spatial correlation feature denotes the i-th column vector of the spatial correlation feature, denotes the i-th column vector of the current frame motion feature denotes the i-th column vector of the current frame motion feature, denotes the normalized spatial correlation coefficient at (i, j), W v' denotes the second trainable value matrix, N denotes the total number of columns of the matrix.

4. The method of claim 3, wherein, The pixels occupied by an object L before and after movement are l and l', the depths are Z and Z', and the pixels occupied by the object are calculated from the camera focal length f and the depth, i.e. l = fL / Z, l' = fL / Z', the relationship between the optical flow scale change and the depth scale change Z' / Z is established; the coordinates of a point in the image coordinate system of the two frames before and after are p and p', the coordinates in the three-dimensional coordinate system are P and P', in combination with the camera intrinsic matrix K, it can be known that ZP = KP, Z'p' = Kp', and the scene flow s of the p point is calculated p = P' - P; the scene flow is calculated pixel by pixel on the frame I t , and the scene flow graph S is obtained o , in combination with the time difference Δt of the current frame I t and the next frame I t+1 , the movement speed s p / Δt is calculated pixel by pixel, and the velocity graph V is obtained o ; Step S400 specifically comprises: wherein, is the motion velocity of any pixel point p on the current frame I t , Δt is the time difference from the current frame to the next frame, is the depth of pixel point p of frame I t+1 , is the depth of pixel point p of frame I t .

5. The method of claim 4, wherein, Step S500 comprises: Step S510: comparing the pixel-by-pixel motion speed with a preset speed threshold, and retaining pixels in the speed map with a motion speed greater than the preset speed threshold; Step S520: segmenting the speed according to a preset segmentation principle, constructing connected regions at different levels, regarding connected regions with a speed difference within a preset range as a whole, outputting the connected regions as a moving target, and regarding the average speed of the connected regions as a motion speed.

6. An unattended system, characterized by The application comprises a wheeled mobile robot, a standard visible light camera, and a computing platform; the standard visible light camera is arranged on the wheeled mobile robot; the wheeled mobile robot advances at a preset speed; the standard visible light camera captures an area in front; and the computing platform is used to execute the method according to any one of claims 1 to 5 to obtain a moving target and a motion speed.

Images