A Fast Detection Method for Distant Small Targets by Fusing Depth Information

Abstract

The present invention belongs to the technical field of target detection using machine vision, and discloses a method for rapid detection of small targets at a distance that integrates depth information, comprising steps 1: preprocessing of sensor information; 2: selection of regions of interest; 3: detection of regions of interest; 4: guidance of detection results based on motion tracking; and 5: combining and outputting detection-tracking results. The present invention uses depth information to estimate the distance between the sensor and the target object and adjusts the detection area, enabling existing detectors to obtain more visual information about the target object. This improves the effective detection distance of the visual detection algorithm for the target object with less computing resources, thereby enhancing the service robot's ability to detect targets that are relatively far away and small in size indoors. Furthermore, the present method utilizes the high spatial consistency of the robot's front and rear image frames during movement, combining the respective advantages of target detection and tracking, thereby improving detection stability while maintaining a high execution speed of the algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection by machine vision, and in particular relates to a method for quickly detecting small targets at a distance by integrating depth information. Background Art

[0002] In mobile robots' object manipulation tasks, target detection technology plays an important role in environmental perception. Although deep learning-based visual target detection technology has continued to develop in recent years, with its precision and recall rates continuing to improve, mobile robots operate in real-world environments. However, changes in visual information caused by non-target objects in the environment and the movement of the robot itself reduce the performance of target detection. In addition, since the target object and the robot's sensor module are sometimes too far apart, directly using this visual information as input to the target detector will result in too much environmental content in the input image and too little visual information about the target object. In this case, it is more difficult for mobile robots to detect target objects in complex indoor environments. In recent years, depth sensors have appeared on an increasing number of robots, but the application of depth information in target detection is still relatively rare.

[0003] Existing vision-based object detection algorithms have relatively poor detection capabilities for small targets (less than 16*16 pixels). Using models with larger input image sizes can improve this detection capability to a certain extent. However, the algorithm's running speed decreases rapidly as the input resolution increases, making it time-consuming to improve the detection capability of distant targets by increasing the model's input resolution. Most RGB-D-based detection algorithms utilize depth information as additional channel information in the algorithm model. These methods offer limited improvement in their utilization of visual information and require additional depth datasets for training. These methods are relatively expensive to implement, and the corresponding improvement in detection performance is limited. Furthermore, these methods utilize depth information in a relatively limited manner and do not exploit the high correlation between similar image frames during mobile robot operation. Summary of the Invention

[0004] The present invention aims to provide a method for quickly detecting small targets at a distance by integrating depth information, so as to solve the above-mentioned technical problems.

[0005] To solve the above technical problems, the specific technical solution of the present invention is a method for quickly detecting small targets at a distance by integrating depth information as follows:

[0006] A method for rapidly detecting small targets at a distance by integrating depth information comprises the following steps:

[0007] Step 1: Perception information preprocessing;

[0008] Step 2: Region of interest selection;

[0009] Step 3: Region of interest detection;

[0010] Step 4: Guidance based on detection results of motion tracking;

[0011] Step 5: Combine and output the detection-tracking results.

[0012] Furthermore, the step 1 includes the following steps:

[0013] Get a data frame containing a color image and a depth information matrix, align the color image and depth information so that the same area of the color image and depth matrix corresponds to the same world coordinates, then resize the color and depth information to the appropriate size and use it as the input data frame for the current time series of the algorithm.

[0014] Furthermore, the step 2 includes the following steps:

[0015] The depth information matrix of the current data frame, the temporal state information, and the detection results of the previous frame are used to select a sub-region of interest of appropriate size and position, and the sub-region information is stored in the sub-region queue. At the same time, a dequeue operation is performed on the sub-region queue, and the input color image is cropped with the dequeued sub-region information, which is used as the input information of the object detector;

[0016] Among them, the depth information matrix is used to estimate the distance between the target object and the sensor to estimate the size of the region of interest; the temporal state information and the previous frame detection results are used to assign the position of the region of interest at different times; the fixed strategy and the active strategy are used in different frames to obtain the region of interest. These two strategies appear in consecutive frames in a preset specific sequence and are selected according to the temporal state of the current frame; among them, the fixed strategy only uses the temporal state information to obtain the position of the region of interest. During the region selection process, the base points of the regions of interest in different temporal states will be assigned to different vertices of the color image. When obtaining the size of the region of interest, the strategy estimates the distance between the target object and the sensor based on the corresponding depth sub-region. The depth sub-region is determined as follows: the horizontal center line and the vertical center line divide the depth information matrix into upper left, upper right, lower left and lower right, which correspond to the regions of interest with the upper left, upper right, lower left and lower right vertices as the base points respectively;

[0017] The active strategy obtains the position of the base point of the region of interest based on the temporal state information and the prior detection information. The region of interest preferentially covers the area where the detection results of the previous frame exist. When obtaining the position of the region of interest, the base point of the region of interest will be assigned to different vertices of the minimum coverage area of the previous frame detection result. When obtaining the size of the region of interest, the base point position is selected as the corresponding vertex of the depth sub-region, and a fixed size is selected as the size of the depth sub-region.

[0018] Furthermore, the step 2 includes the following specific steps:

[0019] Step 2-1: Initialize the sub-region queue: If the current sub-region queue is empty, initialize the sub-region queue; otherwise, do not initialize it.

[0020] Step 2-2: Get the latest sub-region information: Based on the current time series state information, select a fixed strategy or an active strategy to calculate the sub-region information of interest and put the region information into the sub-region queue;

[0021] Step 2-3: Output the current region of interest information: perform a dequeue operation on the sub-region queue and output a region of interest information as the region of interest of the current frame.

[0022] Furthermore, the initialization process of step 2-1 is as follows:

[0023] In each fixed area, a region of interest of appropriate position and size is created as the initialization result of the sub-region queue: the vertices of the upper left, upper right, lower left and lower right regions of the current frame are used as the base point positions of each region of interest; the mean value of the depth sub-region at the corresponding position is used as the distance between the target object in the area and the sensor, and the inverse proportional function is used to calculate the side length of each region of interest based on the relationship between the size and distance of the object seen, and the obtained four region of interest information are placed in the sub-region queue in turn.

[0024] Furthermore, step 3 includes the following steps:

[0025] Use the region of interest information output in step 2 to intercept the corresponding information of the color image, input the intercepted information into an image-based target detection algorithm, and obtain the corresponding detection result.

[0026] Furthermore, step 4 includes the following steps:

[0027] Use motion tracking to estimate the position offset of target objects in adjacent frames and adjust the positions of all target objects detected in previous frames.

[0028] Furthermore, the step 4 includes the following specific steps:

[0029] The sparse optical flow method is applied to the preprocessed color image sequence to obtain the optical flow field of all feature points in the current frame relative to the previous frame image; for each target object in the previous frame color image, the average of the velocity fields of several feature points closest to the bounding box is used as the position offset of the detection result to adjust the position of each object detection result in the color image.

[0030] Furthermore, the step 5 includes the following steps:

[0031] Combine the detection results obtained in step 3 and step 4 as the detection result outputted by the current frame; if there is no new data frame input, terminate the algorithm execution; if there is, go to step 1 and re-execute all steps of the method.

[0032] Furthermore, the combination method of step 5 is as follows:

[0033] For all detection results guided by motion tracking, the results within the region of interest of the current frame are cleared, and then the detection results obtained in step 3 are superimposed with all detection results guided by motion tracking.

[0034] The method for rapidly detecting small targets at a distance by integrating depth information of the present invention has the following advantages:

[0035] 1. This invention uses depth information to estimate the distance between the sensor and the target object and adjust the detection area. This allows existing detectors to obtain more visual information about the target object, increasing the effective detection range of the visual detection algorithm with reduced computing resources. This improves the service robot's ability to detect small and distant objects indoors.

[0036] 2. This invention takes advantage of the high spatial consistency of the robot's front and back image frames during movement, and utilizes the respective advantages of target detection and tracking. Combining the two, it improves the stability of detection while maintaining a high execution speed of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall logic diagram of the present invention.

[0038] Figure 2 This is a processing flow chart in a specific embodiment of the present invention.

[0039] Figure 3 It is a grayscale schematic diagram of the color image and the corresponding pseudo-color image of the depth information matrix.

[0040] Figure 4 Output graph for detection results. DETAILED DESCRIPTION

[0041] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a method for quickly detecting small targets at a distance by integrating depth information of the present invention with reference to the accompanying drawings.

[0042] In order to enhance the detection capability and detection stability of mobile robots for small target objects at a long distance in indoor environments while maintaining good real-time detection, the present invention provides a method: using depth information to estimate the distance between the target object and the sensor, and based on this, cropping the corresponding area of the input image frame so that the detector can obtain richer visual information of the target object; at the same time, utilizing the correlation between similar image frames during the operation of the mobile robot, the detection results between adjacent frames are transmitted through a motion tracking method to improve detection stability, reduce detection tasks, and improve algorithm execution efficiency.

[0043] like Figure 1 As shown, the method for quickly detecting small targets at a distance based on the fusion of depth information and monocular vision information of the present invention comprises the following steps:

[0044] Step 1: Perceptual Information Preprocessing. Obtain a data frame containing a color image and a depth matrix. Align the color image and depth information so that the same regions of the color image and depth matrix correspond to the same world coordinates. Then, resize the color and depth information to the appropriate size and use it as the input data frame for the current time series of the algorithm.

[0045] Step 2: Region of Interest Selection. Using the depth matrix of the current data frame, the temporal state information, and the detection results of the previous frame, a subregion of interest of appropriate size and position is selected and stored in a subregion queue. Simultaneously, the subregion queue is dequeued and the dequeued subregion information is used to crop the input color image and serve as the input to the object detector.

[0046] The depth information matrix is used to estimate the distance between the target object and the sensor, thereby estimating the size of the region of interest. Temporal state information and previous frame detection results are used to assign the location of the region of interest at different times. To leverage this temporal state information and previous frame detection results, this method uses fixed and active strategies to acquire the region of interest across different frames. These two strategies can appear in a specific, pre-set sequence across consecutive frames, with the selection of the strategy based on the temporal state of the current frame.

[0047] Among them, the fixed strategy only uses the temporal state information to obtain the location of the region of interest. During the region selection process, the base points of the region of interest in different temporal states are assigned to different vertices of the color image. When obtaining the size of the region of interest, this strategy estimates the distance between the target object and the sensor based on the corresponding depth sub-region. The depth sub-region is determined as follows: the horizontal center line and the vertical center line divide the depth information matrix into the upper left, upper right, lower left and lower right, corresponding to the region of interest with the upper left, upper right, lower left and lower right vertices as the base points, respectively.

[0048] The active strategy uses temporal state information and prior detection information to determine the location of the base point of the region of interest (ROI). The ROI prioritizes coverage of the area detected by the previous frame. When determining the location of the ROI, the base points are assigned to different vertices in the area with the minimum coverage of the previous frame's detection results. When determining the size of the ROI, the base point locations are selected as the corresponding vertices of the depth sub-region, and a fixed size is selected as the depth sub-region size.

[0049] The specific contents of this step are as follows:

[0050] Step 2-1: Initialize the sub-region queue. If the current sub-region queue is empty, initialize the sub-region queue; otherwise, do not initialize it.

[0051] The initialization process is as follows: Within each fixed region, a region of interest (ROI) of appropriate location and size is created as the initialization result for the sub-region queue. The details are as follows: The vertices of the upper left, upper right, lower left, and lower right regions of the current frame are used as the base points of each ROI. The mean of the depth sub-regions at the corresponding locations is used as the distance between the target object in that region and the sensor. Based on the relationship between the observed object size and distance, the side length of each ROI is calculated using an inverse proportional function. The resulting information for the four ROIs is sequentially placed into the sub-region queue.

[0052] Step 2-2: Get the latest sub-region information. Based on the current time series state information, select a fixed strategy or an active strategy to calculate the sub-region information of interest and put the region information into the sub-region queue.

[0053] Step 2-3: Output the current ROI information. Perform a dequeue operation on the sub-region queue and output a ROI information as the ROI of the current frame.

[0054] Step 3: Detection of Region of Interest: Use the region of interest information output in step 2 to intercept the corresponding information of the color image, and input the intercepted information into an image-based object detection algorithm to obtain the corresponding detection result.

[0055] Step 4: Guided by detection results based on motion tracking. Use motion tracking to estimate the position offset of target objects in adjacent frames and adjust the positions of all target objects in the previous frame detection results.

[0056] The detection-guided process is described below. Sparse optical flow is applied to the preprocessed color image sequence to obtain the optical flow field of all feature points in the current frame relative to the previous frame. For each target object in the previous color image, the average velocity field of the feature points closest to the bounding box is used as the position offset of the detection result to adjust the position of each object detection result in the color image.

[0057] Step 5: Combine and output the detection-tracking results. Combine the detection results from Steps 3 and 4 to output the detection result for the current frame. If no new data frame is input, the algorithm terminates. If so, go back to Step 1 and re-execute all steps of the algorithm.

[0058] The combination method is described as follows: for all detection results guided by motion tracking, the results within the region of interest of the current frame are cleared, and then the detection results obtained in step 3 are superimposed with all detection results guided by motion tracking.

[0059] like Figure 2 As shown, the following is described by way of examples:

[0060] Step 1: Use Realsense d435i to obtain a data frame containing a color image and a depth information matrix, and preprocess it as input data for the algorithm.

[0061] Use Realsense d435i as the sensor to acquire data frames. Each data frame includes a 1280*720 color image and a depth matrix at the same moment. Align the depth matrix with the corresponding color image, perform a 720*720 center crop on the aligned color image and depth matrix, and use the cropped color image fcol and the corresponding depth matrix fdep as input information for the detection algorithm. fcol and fdep are as follows: Figure 3 shown.

[0062] Step 2: In this implementation, the frame sequence number seqf represents the current data frame sequence, serving as temporal state information. The previous frame detection result resp stores the detection results of several previous visual image frames, consisting of several object bounding boxes bi = {cx, cy, w, h, id}, where cx, cy are the coordinates of the bounding box center, w, h are the width and height of the bounding box, and id is the corresponding object type. The subregion queue qroi consists of regions of interest roi = {x0, y0, w, h}, containing up to five RoIs. To estimate distances between different regions, this method uses horizontal and vertical centerlines to divide fdep into four depth subregions. These subregions are divided into upper left, upper right, lower left, and lower right subregions based on their positions. The implementation details are detailed below.

[0063] Step 2-1: If qroi is empty, create four initial RoIs based on the depth sub-region of the current fdep, and input all created RoIs into qroi in sequence to initialize the queue.

[0064] In this implementation, the RoI's position is determined by its base point. Since it is a square, the RoI's size is determined by its side length. When creating the initial RoI, the RoI uses the position of a vertex in fcol as its base point. The specific positions of the RoI's four vertices are then determined based on its side lengths. This is described below.

[0065] For the initial RoIs located in the upper left, upper right, lower left, and lower right, respectively, the upper left, upper right, lower left, and lower right vertices of fcol are used as the corresponding RoI vertices. The side lengths of these RoIs are inversely proportional to the distance between the sensor and the corresponding region. Therefore, the inverse proportional function l = k / d is used to calculate these side lengths. Here, l is the side length, k is the constant of the inverse proportional function, and d is the distance between the sensor and the corresponding target region. d is estimated based on the average depth of the corresponding region. For the upper left RoI, d is estimated based on the upper left subregion of fdep, and similarly for other RoIs. RoI information can be calculated based on the base point position and side lengths of the RoI.

[0066] Step 2-2: Select a fixed or active strategy based on the current fseq to obtain the current RoI information and input it into qroi.

[0067] In this embodiment, the preset sequences of the two strategies fseq are described as follows: the fixed strategy is used when fseq is an integer multiple of 5 or ipre is empty, and the active strategy is used otherwise. Both strategies cyclically select subregions in the order of upper left -> upper right -> lower left -> lower right of the corresponding region.

[0068] When using the fixed strategy, this method selects the fcol vertex corresponding to the current fseq as the ROI's positional base point and calculates the ROI size based on the distance between the sensor and the corresponding fdep subregion. The ROI size calculation method is the same as in step 2-1. For example, if the fixed strategy selects the upper-left subregion under the current fseq, this method selects the upper-left vertex as the current ROI's fixed point, selects the upper-left subregion in fdep to estimate the distance between the sensor and the corresponding region, and calculates the ROI size.

[0069] When using the active strategy, this method first obtains the minimum area Rres covered by the detection results of the previous frame, selects the Rres vertex corresponding to the current fseq as the position base point of the RoI, uses the corresponding area in fdep as the depth information sub-region to estimate the distance between the sensor and the corresponding area, and calculates the ROI size.

[0070] Step 2-3: Perform a dequeue operation on qroi and output roiout = {x0, y0, w, h} as the region of interest of the current color image. x0, y0 are the coordinates of the top left corner of the roi, and w, h are its width and height respectively.

[0071] Step 3: Extract the information fcolroi in the roiout description area from fcol, input fcolroi into the yolov3 detector with Darknet-53 as the backbone network, and obtain the detection result res of fcolroi. res consists of the detection boxes bi of several target objects.

[0072] Step 4: Apply the LK optical flow method to the sparse feature points extracted from the color images fcol0 and fcol1 of the previous and current frames. Adjust the spatial position of the target obtained in resp according to the optical flow field guidance of these feature points, and maintain the corresponding sparse feature point information.

[0073] Optical flow guidance adjusts the position of the target object on the imaging plane by calculating the average optical flow vector corresponding to the five closest feature points. All target objects in resp are guided by optical flow, the position changes of the target objects on the imaging plane are propagated, and resp is updated with the adjusted position information of all targets.

[0074] The method for maintaining the feature points of the visual image frame is as follows: if the number of feature points of fcol1 is less than 50, the feature point extraction method is reused to extract 100 feature points on the color image.

[0075] Step 5: Use step 3 to obtain the detection result of the corresponding area of roi in resp covered by res, and use the covered result resout as the detection output of fcol. If there is no new data frame, the method execution ends; otherwise, go to step 1.

[0076] The detection result overwriting method is as follows: Clear the bi in resp where the cx, cy coordinates are within roi, replace it with the bi in res, and use it as the latest detection result resout, and use it as resp for subsequent detection results. The grayscale image of the output detection color image is as follows Figure 4 As shown, the white bounding box marked with a number in the upper left corner represents the region of interest of the current frame, the white bounding box marked with the target type represents the detection result of the color image of the current frame, and the gray bounding box marked with the target type represents the detection result guided by the prior detection result.

[0077] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A method for rapid detection of small targets at a distance by integrating depth information, characterized in that: The steps include: Step 1: Perception information preprocessing; Get a data frame containing a color image and a depth information matrix, align the color image and depth information so that the same area of the color image and depth matrix corresponds to the same world coordinates, then resize the color image and depth information to the appropriate size and use it as the input data frame for the current time series of the algorithm; Step 2: Region of interest selection; The depth information matrix of the current data frame, the temporal state information, and the detection results of the previous frame are used to select a sub-region of interest of appropriate size and position, and the sub-region information is stored in the sub-region queue. At the same time, a dequeue operation is performed on the sub-region queue, and the input color image is cropped with the dequeued sub-region information, which is used as the input information of the object detector; The depth information matrix is used to estimate the distance between the target object and the sensor, thereby estimating the size of the region of interest. The timing state information and the detection results of the previous frame are used to assign the location of the region of interest at different times. The region of interest is acquired in different frames using a fixed strategy and an active strategy. These two strategies appear in a preset specific sequence in consecutive frames and are selected based on the timing state of the current frame. Among them, the fixed strategy only uses the temporal state information to obtain the position of the region of interest. During the region selection process, the base points of the region of interest in different temporal states will be assigned to different vertices of the color image. When obtaining the size of the region of interest, this strategy estimates the distance between the target object and the sensor based on the corresponding depth sub-region. The depth sub-region is determined as follows: the horizontal center line and the vertical center line divide the depth information matrix into the upper left, upper right, lower left and lower right, which correspond to the regions of interest with the upper left, upper right, lower left and lower right vertices as the base points, respectively; The active strategy obtains the location of the base point of the region of interest based on the temporal state information and the prior detection information. The region of interest preferentially covers the area where the detection results of the previous frame exist. When obtaining the location of the region of interest, the base point of the region of interest will be assigned to different vertices of the minimum coverage area of the previous frame detection result. When obtaining the size of the region of interest, the base point position is selected as the corresponding vertex of the depth sub-region, and a fixed size is selected as the size of the depth sub-region; Step 3: Region of interest detection; Step 4: Guidance based on detection results of motion tracking; Step 5: Combine and output the detection-tracking results.

2. The method for rapid detection of small targets at a distance by integrating depth information according to claim 1, characterized in that: The step 2 includes the following specific steps: Step 2-1: Initialize the sub-region queue: If the current sub-region queue is empty, initialize the sub-region queue; otherwise, do not initialize it. Step 2-2: Get the latest sub-region information: Based on the current time series state information, select a fixed strategy or an active strategy to calculate the sub-region information of interest and put the region information into the sub-region queue; Step 2-3: Output the current region of interest information: perform a dequeue operation on the sub-region queue and output a region of interest information as the region of interest of the current frame.

3. The method for rapid detection of small targets at a distance by integrating depth information according to claim 2, characterized in that: The initialization process of step 2-1 is as follows: Create a region of interest with a suitable position and size in each fixed area as the initialization result of the sub-area queue: use the vertices of the upper left, upper right, lower left, and lower right areas of the current frame as the base point positions of each region of interest; The mean value of the depth sub-region at the corresponding position is used as the distance between the target object in the region and the sensor. According to the relationship between the size and distance of the object seen, the inverse proportional function is used to calculate the side length of each region of interest, and the obtained information of the four regions of interest are placed in the sub-region queue in turn.

4. The method for rapid detection of small targets at a distance by integrating depth information according to claim 1, characterized in that: The step 3 comprises the following steps: Use the region of interest information output in step 2 to intercept the corresponding information of the color image, input the intercepted information into an image-based target detection algorithm, and obtain the corresponding detection result.

5. The method for rapid detection of small targets at a distance by integrating depth information according to claim 1, characterized in that: The step 4 comprises the following steps: Use motion tracking to estimate the position offset of target objects in adjacent frames, guiding the adjustment of the positions of all target objects detected in previous frames.

6. The method for rapid detection of small targets at a distance by integrating depth information according to claim 1, characterized in that: The detection result guidance process of step 4 is as follows: The sparse optical flow method is applied to the preprocessed color image sequence to obtain the optical flow field of all feature points in the current frame relative to the previous frame image; for each target object in the previous frame color image, the average of the velocity fields of several feature points closest to the bounding box is used as the position offset of the detection result to adjust the position of each object detection result in the color image.

7. The method for rapid detection of small targets at a distance by integrating depth information according to claim 1, characterized in that: The step 5 comprises the following steps: Combine the detection results obtained in step 3 and step 4 as the detection result outputted by the current frame; if there is no new data frame input, terminate the algorithm execution; if there is, go to step 1 and re-execute all steps of the method.

8. The method for rapid detection of small targets at a distance by integrating depth information according to claim 1, characterized in that: The combination method of step 5 is as follows: For all detection results guided by motion tracking, the results within the region of interest of the current frame are cleared, and then the detection results obtained in step 3 are superimposed with all detection results guided by motion tracking.

Images