Target detection and positioning method based on deep optical flow and YOLOv3 spatio-temporal fusion

Abstract

The application discloses a target detection and positioning method of deep optical flow and YOLOv3 space-time fusion, motion information of a human foot target in a time domain and position information of the human foot target in a space domain are respectively extracted through a FlowNet2 and a YOLOv3 network to perform foot detection, then according to a fusion strategy established, the space-time information of the foot target extracted by the two networks is fused, so that the foot target positioning is realized. The application solves the problem that in the prior art, when the foot is static, the target is small, there is occlusion, the background light changes greatly and the foot shape changes greatly, the human foot cannot be accurately detected and positioned.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot vision detection method, and particularly relates to a target detection and positioning method of deep optical flow and YOLOv3 spatiotemporal fusion. BACKGROUND

[0002] With the continuous development of artificial intelligence and robot technology, intelligent service robots are increasingly applied in daily production and life. Human following robots belong to intelligent service robots, which can help people transport goods or track specific targets. Human target detection and positioning algorithm with strong robustness is a prerequisite for service following robots to complete following, which represents the development direction of intelligent service robots.

[0003] Intelligent service robots are equipped with depth cameras to detect, identify and locate video targets. Small intelligent service robots have a low height, and the depth camera cannot collect the upper body of the human in a short distance. In this case, it is reasonable to detect the human foot as the target. Although the classic optical flow algorithm (such as Lucas-Kanade) can extract the motion information of the foot in the time domain, the motion information is incomplete, the foot target edge is easily missing, and it is greatly affected by light changes, camera motion and pedestrian foot turning, which brings great difficulty to the foot target positioning. The optical flow neural network based on deep learning estimates the optical flow of the foot through the pre-trained optical flow model, and the optical flow estimation result is more accurate than the classic optical flow algorithm. However, the problems of foot missing and false detection caused by foot stillness, small target, occlusion and severe background light changes have not been well solved. YOLOv3 network can solve these problems. The non-maximum suppression algorithm effectively improves the missing detection phenomenon of occlusion, the model trained by the foot real label file solves the missing and false detection problems caused by severe light changes, and the algorithm also increases the detection ability of different scale targets. However, the YOLOv3 algorithm also has disadvantages. When the human motion causes the foot shape to change greatly (heel lifting, shoe sole becoming the target), it is no longer the original foot shape, resulting in missing detection. SUMMARY

[0004] The purpose of the present application is to provide a target detection and positioning method of deep optical flow and YOLOv3 spatiotemporal fusion, which solves the problems of foot missing and false detection caused by foot stillness, small target, occlusion, severe background light changes and large foot shape changes in the existing human foot detection process.

[0005] The technical scheme adopted by the present application is: a target detection and positioning method of deep optical flow and YOLOv3 spatiotemporal fusion, comprising the following steps:

[0006] Step 1, collect human walking videos under different scenes and convert them into video sequences;

[0007] Step 2, make a human walking training set from the video sequences obtained in step 1.

[0008] Step 3, label the training set obtained in step 2 to obtain a training set label file.

[0009] Step 4, build a YOLOv3 deep learning framework.

[0010] Step 5, train the YOLOv3 deep learning framework obtained in step 4 using the label file obtained in step 3 to obtain a human foot target detection weight model.

[0011] Step 6, build a FlowNet2 deep optical flow network framework.

[0012] Step 7, train the FlowNet2 deep optical flow network framework obtained in step 6 using the training set obtained in step 2 to obtain a human walking optical flow model.

[0013] Step 8, collect human walking videos under complex scenes as a test set for human foot target detection.

[0014] Step 9, input the test set obtained in step 8 into the human walking optical flow model obtained in step 7 to obtain foot temporal motion information, and visualize the foot temporal motion information through color coding.

[0015] Step 10, position the foot temporal motion information obtained in step 9, and input the test set obtained in step 8 into the human foot target detection weight model obtained in step 5 to obtain foot spatial position information, and fuse the foot space-time information to obtain the foot target position.

[0016] The characteristics of the present application are that,

[0017] In step 2, the video sequences are renamed according to the format "00001.jpg" to make a human walking training set.

[0018] In step 3, the training set is labeled using LabelImg tool.

[0019] The complex scenes in step 8 include conditions such as light changes, background and object interference, target and camera motion, target scale changes and target occlusion.

[0020] Step 9 specifically includes the following steps: first, input the continuous two frames of images to be solved, image 1 and image 2, into the FlowNetC network, and obtain the optical flow field image, image 1 and image 2 after convolution network solving, and then use the optical flow estimated by the upper layer network to obtain the continuous image and brightness difference by warping transformation of image 2; second, take the optical flow field predicted by each group of images as input, fuse the results of large and small displacement networks through the fusion network, and then perform convolution operation to obtain the optical flow estimation result; finally, color coding is performed to obtain the foot target optical flow visualization map.

[0021] Step 10 specifically includes the following steps:

[0022] Step 10.1, foot time domain motion information positioning processing

[0023] The optical flow visualization map is binarized, and at the same time, the maximum connected region is obtained by morphological operation and the minimum values x1 and y1 of the left and upper boundaries and the maximum values x2 and y2 of the right and lower boundaries are calculated to obtain the boundary box (x1, y1, x2-x1, y2-y1) of the foot target;

[0024] Step 10.2, foot spatial position information acquisition

[0025] The foot RGB image is input into the YOLOv3 network at the same time, and the foot target boundary box is automatically output by the boundary box acquisition code written;

[0026] Step 10.3, foot space-time information fusion

[0027] When YOLOv3 and FlowNet2 detect the foot target, first, the detection result is positioned to obtain the boundary box position, and then the minimum values of the horizontal coordinates of the two foot boundary boxes are compared, and the smaller one is the left foot. The left foot and the right foot are fused by formula (1):

[0028]

[0029] In formula (1), Bbox Fuse is the fusion boundary box of the foot, Bbox Yolov3 is the foot boundary box detected by YOLOv3, and Bbox Flow is the foot boundary box detected by FlowNet2. The left, upper, right and lower coordinates of the boundary box are fused by mean fusion during fusion;

[0030] If YOLOv3 does not predict the foot, the prediction result of FlowNet2 is taken as the final prediction result; if FlowNet2 does not predict the foot, the prediction result of YOLOv3 is taken as the final prediction result.

[0031] The depth optical flow and YOLOv3 spatiotemporal fusion target detection and positioning method has the advantages that the target spatiotemporal information fusion strategy is adopted to solve the problem that the human foot cannot be accurately detected and positioned when the foot is static, the target is small, there is occlusion, the background light changes greatly, and the foot shape changes greatly, and the foot target detection accuracy is improved, laying a foundation for foot tracking research. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a FlowNet2 network structure diagram;

[0033] Figure 2 is a YOLOv3 network structure diagram;

[0034] Figure 3 is a F-YOLOv3 network structure diagram;

[0035] Figure 4 is a foot target fusion positioning strategy diagram;

[0036] Figure 5a ) and Figure 5b ) are respectively accuracy-recall rate curve diagrams of YOLOv3 algorithm and F-YOLOv3 algorithm;

[0037] Figure 6a ) and Figure 6b ) are respectively accuracy and recall rate bar charts of different algorithms under five video scenes;

[0038] Figure 7 is a foot detection and positioning effect diagram of different algorithms;

[0039] Figure 8a ) and Figure 8b ) are respectively foot detection result diagrams and foot detection scene diagrams of the robot when a single person walks;

[0040] Figure 9a ) and Figure 9b ) are respectively foot detection result diagrams and foot detection scene diagrams of the robot when a single person walks; DETAILED DESCRIPTION

[0041] The application will be described in detail below in combination with the drawings and specific embodiments.

[0042] The application provides a depth optical flow and YOLOv3 spatiotemporal fusion target detection and positioning method, which is implemented according to the following steps.

[0043] Step 1, collect human walking videos under different scenes, and convert the human walking videos into video sequences.

[0044] Step 2, rename the video sequence according to the format of "00001.jpg", and make a human walking training set.

[0045] Step 3, label the training set using LabelImg tool to obtain the training set label file; LabelImg annotation tool saves XML file, and conversion code is written to convert XML format to YOLO data format.

[0046] Step 4, build YOLOv3 deep learning framework, specific environment: Windows10, Tensorflow2.0, Python3.7.

[0047] Step 5, use the training set label file in step 3 to train YOLOv3 deep learning framework to obtain human foot target detection weight model.

[0048] Step 6, build FlowNet2 deep optical flow network framework, specific environment: Windows10, Pytorch0.4.1.

[0049] Step 7, use the training set in step 2 to train FlowNet2 network to obtain human walking optical flow model.

[0050] Step 8, use a, b, c, d, e five kinds of complex scene human walking video as test set for testing, complex scene includes light change, background and object interference, target and camera common movement, target scale change, target occlusion and other conditions, specific scene description is shown in the following table 1.

[0051] Step 9, input the test set into the FlowNet2 network to obtain the foot time domain motion information, and visualize the motion information by color coding.

[0052] FlowNet network is divided into shrinkage and expansion two parts, the network input is two frames of images, the shrinkage part is responsible for compressing the available information, and the expansion part is responsible for restoring the image and optical flow feature map to the original image size. The shrinkage part has two structures of FlowNetS and FlowNetC. FlowNetS directly convolves the input image 9 times to obtain the optical flow feature map, while FlowNetC convolves two frames of images to obtain the optical flow feature, then the Feature map is matched through Correlation layer, and finally the convolution is continued to obtain more complete Feature map.

[0053] FlowNet2 is obtained by stacking FlowNetS and FlowNetC, and the specific structure is as follows Figure 1Small displacement is small displacement sub-network, and the network based on this data set has good effect on small displacement movement of the target in the real world. Wrap operation can not only make the positioning more accurate, but also prevent overfitting caused by network stacking. FlowNetC-FlowNetS-FlowNetS is connected in series to form a large displacement solving network, and FlowNet-SD network focuses on small displacement solving.

[0054] When using FlowNet2 to estimate the optical flow of the foot, first input two consecutive images 1 and 2 to be solved into the FlowNetC network, and then obtain the optical flow field image, image 1 and image 2 after convolution network solving. The optical flow estimated by the upper layer network is used to transform the image 2 to obtain the continuous image and brightness difference. Secondly, the optical flow field predicted by each group of images is taken as input, and the results of large and small displacement networks are fused through the fusion network, and then convolution operation is performed to obtain the optical flow estimation result. Finally, color coding is performed to obtain the foot target optical flow visualization map.

[0055] Step 10, input the test set into the YOLOv3 network to obtain the spatial position information of the foot, and at the same time, the temporal motion information obtained in step 9 is positioned to obtain the position of the foot target, and the positions of steps 9 and 10 are averaged to obtain the final position of the foot target. The specific implementation is as follows:

[0056] Step 10.1, foot temporal motion information positioning processing. The color depth of the optical flow visualization map represents the foot movement speed, and the color type represents the foot movement direction. In order to obtain the foot target bounding box, the optical flow visualization map is binarized, and at the same time, the maximum connected region is obtained by using morphological operation and the minimum values x1 and y1 of the left and upper boundaries and the maximum values x2 and y2 of the right and lower boundaries are calculated, that is, the bounding box of the foot target (x1, y1, x2-x1, y2-y1) is obtained.

[0057] Step 10.2, foot spatial position information acquisition. The foot RGB image is input into the YOLOv3 network at the same time, and the bounding box acquisition code algorithm written will automatically output the foot target bounding box. The basic principle of YOLOv3 algorithm is to use a fixed size image as network input, and use the idea of regression to obtain the position of the bounding box and the target class, and its network structure is as shown in Figure 2

[0058] ​The feature extraction network of YOLOv3 is Darknet-53, and the convolution kernel of 1*1 and 3*3 size is alternately used for convolution. The full connection layer has three different outputs of large, medium and small, which can increase the adaptability of different scale targets. The network can greatly increase the network depth by referring to the residual network idea. The network input is an RGB image, which divides the image into S*S cells, each cell is responsible for detecting the target whose center position falls into it, each cell contains B bounding boxes and bounding box confidence c, the prediction output has five dimensions, that is (x, y, w, h, c), and contains C target categories, so the final output size of the network is S*S*(B*5+C), the confidence c=P*IOU, wherein P is the category probability of the grid containing the target, and the value is 0 or 1, and IOU is the intersection over union of the predicted box and the real box. After obtaining the prediction box, non-maximum suppression is completed to complete foot detection.

[0059] In order to realize the foot target detection of YOLOv3, the present application constructs a plurality of human foot walking training sets in different scenes, uses LabelImg tool to complete label making, and the foot walking model is trained in Nvidia GeForce 920M graphics card. Because the training set is a self-built data set, the original prior box is no longer applicable, so the K-means clustering algorithm is used to obtain 9 prior box sizes suitable for foot targets. The maximum number of iterations of model training is 200, and the loss function value of the training process finally converges to about 1.65.

[0060] Step 10.3, foot space-time information fusion. In order to solve the problems of foot missing and false detection caused by deep optical flow in the scene of foot static, small target, occlusion and severe background light change, and the problem of poor adaptability of YOLOv3 detection when the foot shape changes greatly, the present application adopts F-YOLOv3 network to fuse the foot space-time information to realize foot target detection, and the network structure is as shown in Figure 3 .

[0061] The foot target fusion positioning strategy as shown in Figure 4 is proposed. Because of the particularity of the foot target, it is necessary to judge whether the detection result is the left foot or the right foot before fusing the foot space-time information. When YOLOv3 and FlowNet2 detect the foot target, the boundary box position of the detection result is obtained by positioning, and then the minimum value of the horizontal coordinates of the boundary boxes of the two feet is compared, and the smaller one is the left foot. At this time, the space-time information of the left foot and the right foot can be fused, and the fusion strategy is as shown in formula (1).

[0062]

[0063] In formula (1), Bbox FuseBbox is a fusion bounding box of feet Yolov3 Bbox is a bounding box of feet detected by YOLOv3 Flow Bbox is a bounding box of feet detected by FlowNet2, and the left, upper, right and lower coordinates of the bounding box are fused at the same time when fusing.

[0064] If YOLOv3 does not predict feet, the prediction result of FlowNet2 will be the final prediction result. Similarly, if FlowNet2 does not predict feet, the prediction result of YOLOv3 will be the final prediction result. After fusion, the position of the fused foot target bounding box is obtained through positioning, which lays a foundation for the initialization of the target position in the future research of foot tracking.

[0065] The present application shoots a total of 5 groups of human walking videos in different scenes indoors and outdoors as test sets, and the video scene description is shown in Table 1.

[0066] Table 1 Scene description of five groups of actually shot pedestrian walking videos

[0067]

[0068] The accuracy-recall rate is an important index for measuring the effect of target detection, and the calculation formula is as follows:

[0069]

[0070] In the formula, TP is the number of feet correctly identified by the algorithm, FP is the number of feet incorrectly identified by the algorithm, and FN is the number of feet not identified by the algorithm. The foot detection accuracy-recall rate curves of YOLOv3 and F-YOLOv3 algorithms are shown in Figure 5a and Figure 5b respectively, which are the accuracy-recall rate data of the two algorithms in different video scenes, and the accuracy column chart when the recall rate is 0.8 and the recall column chart when the accuracy is 0.8 are shown in Figure 6a and Figure 6b respectively.

[0071] As can be seen from FIGS. 5 and 6, for the foot target detection of the video d in which the pedestrian turns and the foot is blocked outdoors, when the recall rate is 0.8, the accuracy rate of YOLOv3 is 0.82, and the accuracy rate of F-YOLOv3 is 0.85. When the accuracy rate is 0.8, the recall rate of YOLOv3 is 0.86, and the recall rate of F-YOLOv3 is 0.89. For the video e in which the pedestrian and the camera are both moving and the light changes significantly outdoors, when the recall rate is 0.8, the accuracy rate of YOLOv3 is 0.88, and the accuracy rate of F-YOLOv3 is 0.90. When the accuracy rate is 0.8, the recall rate of YOLOv3 is 0.82, and the recall rate of F-YOLOv3 is 0.84. This shows that the F-YOLOv3 algorithm can still obtain a high accuracy rate and a high recall rate in the complex scene in which the pedestrian turns and the foot is blocked outdoors and the light changes significantly, and the fusion effect is better than that of the YOLOv3 algorithm. In the other scenes a, b and c, the accuracy rate-recall rate of F-YOLOv3 is also higher than that of the YOLOv3 algorithm, which fully shows that the accuracy rate and the recall rate of the F-YOLOv3 algorithm are positively correlated, which conforms to the general trend of the accuracy rate-recall rate curve, and the algorithm has strong robustness.

[0072] The AP is the detection accuracy of each target class, that is, the average accuracy of all recall rates between 0 and 1. The average AP, that is, the foot detection mAP value, can be obtained by averaging the AP of each target class, and the calculation formula is shown as formula (3). Since the foot detection of the present application only contains one target class, the AP value is equal to the mAP value in value.

[0073]

[0074] Table 2 foot detection mAP values of different algorithms

[0075]

[0076] In Table 2, the average mAP value of FlowNet2 under 5 groups of videos is 13.20%, and the mAP value under the complex scene of videos d and e is only about 5%, because the motion displacement of one foot is small when the human body walks, which makes it difficult to detect the time domain motion information, and the background light changes dramatically and the distance between the two feet is too close when the foot turns and is blocked, which causes the time domain motion information to overlap, which also causes the detection to miss and false detection. For the YOLOv3 algorithm, the ground shadow in video e alternates between light and dark, and the camera and the foot move together, causing the foot brightness and shape to change, and the mAP value is the lowest among the 5 groups of videos, with an average mAP of 81.59%. The Max-Fuse algorithm uses a maximum fusion strategy to fuse the spatio-temporal information of the foot based on the inspiration of F-YOLOv3, and the average mAP reaches 84.21%. The F-YOLOv3 algorithm effectively improves the foot miss and false detection phenomenon under the human body turning, occlusion, dramatic light changes and camera motion in videos d and e by using an average fusion strategy for spatio-temporal information of the foot, and the average mAP value reaches 85.48%, which shows that F-YOLOv3 can still achieve good detection effect in complex scenes. In addition, the foot detection mAP value of F-YOLOv3 in videos a-e is close to or higher than that of the comparative algorithm, which proves that the fusion algorithm for foot detection and positioning has better effect.

[0077] To analyze the real-time performance of the algorithm, we compared the FPS of the algorithms listed in Table 2 on an NVIDIA GeForce 1080Ti graphics card. The FPS of the F-YOLOv3 fusion algorithm is 23, which can meet the real-time demand of target detection.

[0078] To demonstrate the robustness of the F-YOLOv3 fusion algorithm for foot target detection and positioning, we compared the foot target detection and positioning effects of different scenes using FlowNet2, YOLOv3 and F-YOLOv3 algorithms, and the effect diagram is shown in Figure 7

[0079] Figure 7 ​The RGB image frames, the foot temporal motion information, the foot detection results of FlowNet2, YOLOv3 and F-YOLOv3 from left to right in the vertical direction, and the foot detection results of five walking scenes from top to bottom in the horizontal direction, including an indoor corridor, an outdoor garden, an outdoor complex background, a foot occlusion and a road with strong light changes. As shown in the figure, when the foot shape changes greatly (heel lifting), the YOLOv3 algorithm will miss detection, while the F-YOLOv3 algorithm fuses the foot temporal motion information and spatial position information, and finally can accurately detect the foot, as shown in the first, second and fifth rows of the figure. When the foot is completely stationary, the FlowNet2 algorithm cannot detect the motion information, resulting in missed detection, while the F-YOLOv3 algorithm does not miss detection, and the fused bounding box is more complete than the YOLOv3 algorithm, as shown in the third row of the figure. When the foot is occluded, the foot temporal motion information extracted by the FlowNet2 algorithm is not easy to distinguish and even occluded, resulting in the detection result containing only one foot, while the F-YOLOv3 algorithm can accurately distinguish and detect the human foot, as shown in the fourth row of the figure. The results show that the F-YOLOv3 algorithm can not only detect the human foot in complex scenes such as turning, occlusion, camera motion and strong background light changes, but also has better fusion effect, which shows that the algorithm has certain robustness in human foot detection.

[0080] To verify the feasibility of the application of the application in mobile robot foot target detection, a Turtlebot robot and ROS are used to complete the test design. The Kinect is used to collect video information in real time, the frame rate is 27 frames / s, the computer model is Intel i5-5200U, the frequency is 2.4Ghz, the system is Ubuntu 16.04 LTS+Kinetic, the experimental environment is divided into indoor and outdoor, the indoor is the corridor and the entrance hall of the teaching building, and the light is bright and stable; the outdoor is the roadside and the parking lot, and the light changes greatly. The walking mode of the pedestrian is divided into pedestrian stationary, vertical direction walking, 45-degree angle with the horizontal direction and multiple people walking together.

[0081] Figure 8a ) is the foot detection result figure of the robot view when a person walks, from left to right, the pedestrian stationary, the pedestrian walking along the horizontal direction at a 45-degree angle, and the foot detection result figure of the pedestrian walking along the vertical direction; Figure 9a ) from left to right, the robot view foot detection result figure of the pedestrian walking at a 45-degree angle with the horizontal line (occluded to different degrees), and multiple people walking together. Combined with Figure 8b ) and Figure 9bIt can be known that the algorithm of the application can accurately detect the human foot target in the scenes of the pedestrian being static, the pedestrian walking along the vertical direction, the pedestrian walking in the direction of 45 degrees with the horizontal line and the foot being shielded, and has the universality of foot target detection in the scene of two people walking together, thereby verifying the reliability of the application of the algorithm of the application to the mobile robot and laying a foundation for the robust target tracking.

Claims

1. A target detection and localization method based on the spatiotemporal fusion of depth optical flow and YOLOv3, characterized in that, Includes the following steps: Step 1: Collect videos of human walking in different scenarios and convert them into video sequences; Step 2: Create a human walking training set from the video sequences obtained in Step 1; Step 3: Label the training set obtained in Step 2 to obtain the training set label file; Step 4: Build the YOLOv3 deep learning framework; Step 5: Use the label file obtained in Step 3 to train the YOLOv3 deep learning framework obtained in Step 4 to obtain the human foot target detection weight model; Step 6: Build the FlowNet2 deep optical flow network framework; Step 7: Use the training set obtained in Step 2 to train the FlowNet2 deep optical flow network framework obtained in Step 6 to obtain the human walking optical flow model. Step 8: Collect human walking videos in complex scenes as a test set for human foot target detection. Complex scenes include conditions such as: lighting changes, background and object interference, target and camera moving together, target scale changes, and target occlusion. Step 9: Input the test set obtained in Step 8 into the human walking optical flow model obtained in Step 7 to obtain foot temporal motion information, and visualize the foot temporal motion information through color encoding; specifically: First, input two consecutive frames of images 1 and 2 to be solved into the FlowNetC network. After the convolutional network solves, it obtains the optical flow field image, image 1, and image 2. The optical flow estimated by the previous layer network is used to distort image 2 to obtain a continuous image and brightness difference; Second, the optical flow field predicted by each set of images is used as input. The results of the large and small displacement networks are fused through the fusion network, and then a convolution operation is performed to obtain the optical flow estimation result; Finally, it is color encoded to obtain a visual image of the foot target optical flow. Step 10: Perform localization processing on the foot temporal motion information obtained in Step 9, and simultaneously input the test set obtained in Step 8 into the human foot target detection weight model obtained in Step 5 to obtain foot spatial location information. Then, perform mean fusion on the foot spatiotemporal information to obtain the foot target location. Specifically, the following steps are included: Step 10.1: Foot temporal motion information localization processing The optical flow visualization is binarized, and morphological operations are used to obtain the largest connected region and calculate the minimum values ​​of its left and upper boundaries. , and the maximum value of the right and bottom boundaries , Obtain the bounding box of the foot target. ; Step 10.2: Obtaining Foot Spatial Position Information The RGB image of the foot is simultaneously input into the YOLOv3 network, and the bounding box of the foot target is automatically output through the written bounding box acquisition code; Step 10.3: Foot spatiotemporal information fusion When YOLOv3 and FlowNet2 detect a foot target, they first locate the detection result to obtain its bounding box position, and then compare the minimum horizontal coordinate of the bounding boxes of the two feet. The smaller value is the left foot. The spatiotemporal information of the left and right feet is fused using formula (1): （1） In equation (1), For the fusion bounding box of the foot, The bounding box of the foot detected by YOLOv3. For the foot bounding boxes detected by FlowNet2, the left, top, right, and bottom coordinates of the bounding boxes are simultaneously fused using the mean. If YOLOv3 does not predict the foot, the prediction result of FlowNet2 is used as the final prediction result; if FlowNet2 does not predict the foot, the prediction result of YOLOv3 is used as the final prediction result.

2. The target detection and localization method based on the spatiotemporal fusion of depth optical flow and YOLOv3 as described in claim 1, characterized in that, In step 2, the video sequence is renamed according to the format "00001.jpg" to create a human walking training set.

3. The target detection and localization method based on the spatiotemporal fusion of depth optical flow and YOLOv3 as described in claim 1, characterized in that, In step 3, the LabelImg tool is used to label the training set.

Images