Image processing method, device and robot vision odometer

Abstract

The application discloses an image processing method and device and a robot vision odometer, and relates to the technical field of computer vision. The technical points of the application comprise: acquiring an RGB image sequence and a depth image sequence containing a target; processing the RGB image sequence and the depth image sequence to acquire a target depth extraction image sequence; screening the target in the target depth extraction image sequence by using a mask extraction algorithm based on K-means clustering to obtain a target binary mask sequence; and processing an image sequence collected by an upper camera of a robot according to the above process to determine the position and posture of the robot. The application has extremely fast calculation speed and can achieve better effect than instance segmentation. Compared with the traditional method, the application has great improvement in positioning accuracy and robustness in a dynamic scene, and has smaller root mean square error.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an image processing method, apparatus, and robot visual odometry. Background Technology

[0002] Maintaining robustness in dynamic environments is a challenge for robot vision SLAM systems. Although SLAM systems have been successfully demonstrated in specific situations, the estimation of state variables can be affected by dynamically moving objects in the environment, even causing the SLAM system to fail to track them. In vision-based SLAM systems, a person walking dynamically in the environment enters the system through a sensor, and the vision front end extracts all feature points from the image without judgment, including non-static feature points generated by the moving human body. The system adds these feature points to the back-end optimization. As the human body moves, the feature points that should be static undergo a huge shift, which makes the pose solved by the entire system inaccurate, resulting in a rapid decrease in the robustness of the system.

[0003] In existing technologies, scholars have conducted specific research from different perspectives. For example, Runz et al. proposed MaskFusion in "Maskfusion: Real-time recognition, tracking and reconstruction of multiple moving objects," which uses an RGB-D camera and instance-level semantic segmentation technology to generate semantic masks for moving objects, thereby achieving object detection and constructing an object-level world map representation. Another example is Yu et al.'s "DS-SLAM: A semantic visual SLAM towards dynamic environments," which proposes a robust semantic visual DS-SLAM for dynamic environments. This system introduces a semantic segmentation network model, masks dynamic feature points through object masking, and uses motion consistency detection methods for further detection. Through these two layers of filtering, dynamic feature points are masked, thereby improving the accuracy of pose solving in dynamic environments. However, the above methods share a common and important problem: the systems cannot run in real time. Summary of the Invention

[0004] In view of the above problems, the present invention proposes an image processing method, a device and a robot visual odometry.

[0005] According to one aspect of the present invention, an image processing method is proposed, the method comprising:

[0006] Obtain the RGB image sequence and depth map sequence containing the target;

[0007] The RGB image sequence and the depth map sequence are processed to obtain a target depth extraction map sequence;

[0008] The target is selected from the target depth extraction map sequence using a mask extraction algorithm based on K-means clustering, and a binary mask sequence of the target is obtained.

[0009] Furthermore, after obtaining the target binary mask sequence, dynamic feature points are removed to obtain the final segmentation mask extraction map.

[0010] Further, processing the RGB image sequence and the depth map sequence to obtain the target depth extraction map sequence includes: using a target detection model to detect targets in the RGB image sequence and obtaining detection boxes; segmenting the targets in the detection boxes from the depth map sequence based on the detection boxes to obtain the target depth extraction map sequence.

[0011] Furthermore, the target selection algorithm based on K-means clustering is used to filter out targets from the target depth extraction map sequence, including:

[0012] The target depth extraction map is judged by the window size. If the window is smaller than a preset threshold, the maximum and minimum depth values ​​of the target depth extraction map are directly output. If the window is not smaller than the preset threshold, the 0 values ​​are removed, and the two-dimensional depth map is converted into one-dimensional data, where the elements in the one-dimensional data are defined as depth samples.

[0013] Perform three-class classification on one-dimensional data. Set the number of clusters in the K-means clustering algorithm to 3. Perform K-means clustering and obtain the index of the smallest centroid cluster and the index of the second smallest centroid cluster from the clustering results.

[0014] Compare the sample size of the smallest centroid cluster and the second smallest centroid cluster, define the cluster with the larger sample size as the moving cluster, and calculate the standard deviation of the moving cluster;

[0015] If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than the preset percentage of the total number of samples, then the maximum and minimum depth values ​​in the motion cluster will be output.

[0016] If the standard deviation of the motion cluster is greater than the empirical value or the number of samples in the motion cluster is less than the preset percentage of the total number of samples, then K-means clustering is performed again with the number of clusters set to 2. The resulting minimum centroid cluster is defined as the motion cluster, and the maximum and minimum depth values ​​in the minimum centroid cluster are output.

[0017] Furthermore, after obtaining the target binary mask sequence, the removal of dynamic feature points includes: generating a mask pyramid based on the image pyramid; marking feature points within the mask range according to the mask pyramid, and removing dynamic feature points by preventing the generation of map points from these feature points.

[0018] According to another aspect of the present invention, an image processing apparatus is provided, the apparatus comprising:

[0019] An image sequence acquisition module is configured to acquire an RGB image sequence and a depth map sequence containing the target;

[0020] The target depth map extraction module is configured to process the RGB image sequence and the depth map sequence to obtain a target depth extraction map sequence;

[0021] The target segmentation mask extraction module is configured to use a mask extraction algorithm based on K-means clustering to filter out targets from the target depth extraction map sequence and obtain a target binary mask sequence.

[0022] Furthermore, it also includes a dynamic feature point removal module, which is configured to generate a mask pyramid based on the image pyramid after obtaining the target binary mask sequence; according to the mask pyramid, feature points within the mask range are marked, and dynamic feature points are removed by preventing the generation of map points by the feature points, so as to obtain the final segmentation mask extraction map.

[0023] Furthermore, the target segmentation mask extraction module uses a K-means clustering-based mask extraction algorithm to filter out targets from the target depth extraction map sequence, including:

[0024] The target depth extraction map is judged by the window size. If the window is smaller than a preset threshold, the maximum and minimum depth values ​​of the target depth extraction map are directly output. If the window is not smaller than the preset threshold, the 0 values ​​are removed, and the two-dimensional depth map is converted into one-dimensional data, where the elements in the one-dimensional data are defined as depth samples.

[0025] Perform three-class classification on one-dimensional data. Set the number of clusters in the K-means clustering algorithm to 3. Perform K-means clustering and obtain the index of the smallest centroid cluster and the index of the second smallest centroid cluster from the clustering results.

[0026] Compare the sample size of the smallest centroid cluster and the second smallest centroid cluster, define the cluster with the larger sample size as the moving cluster, and calculate the standard deviation of the moving cluster;

[0027] If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than the preset percentage of the total number of samples, then the maximum and minimum depth values ​​in the motion cluster will be output.

[0028] If the standard deviation of the motion cluster is greater than the empirical value or the number of samples in the motion cluster is less than the preset percentage of the total number of samples, then K-means clustering is performed again with the number of clusters set to 2. The resulting minimum centroid cluster is defined as the motion cluster, and the maximum and minimum depth values ​​in the minimum centroid cluster are output.

[0029] According to another aspect of the present invention, a robot visual odometry is provided, wherein the visual odometry uses the image processing method described above to process image sequences captured by a camera on a robot to determine the robot's position and orientation.

[0030] Furthermore, in the target detection thread that uses the target detection model to detect targets in the RGB image sequence, the front-end visual odometry uses the detection results of the previous frame from the target detection thread.

[0031] The beneficial technical effects of this invention are:

[0032] This invention proposes an image processing method, apparatus, and robot visual odometry. First, an RGB image sequence and a depth map sequence containing the target are acquired. The RGB image sequence and the depth map sequence are then processed to obtain a target depth extraction map sequence. A mask extraction algorithm based on K-means clustering is used to filter out the target from the target depth extraction map sequence, obtaining a target binary mask sequence. The image sequence acquired by a camera on the robot is then processed according to the above process to determine the robot's position and orientation. This invention offers extremely fast computation speed, achieving better results than instance segmentation.

[0033] Experiments were conducted on publicly available datasets to verify the invention. Through both qualitative and quantitative analysis, it was concluded that, compared with traditional methods, the invention significantly improves the accuracy and robustness of localization in dynamic scenes. Furthermore, compared with other methods, the algorithm of the invention has a smaller root mean square error, demonstrating the superiority of the algorithm. Attached Figure Description

[0034] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the detailed description below are incorporated in and form part of this specification, and serve to further illustrate preferred embodiments of the invention and explain the principles and advantages of the invention.

[0035] Figure 1 This is a schematic diagram of the process of an embodiment of the present invention.

[0036] Figure 2 This is a flowchart of the YOLOv5 thread program in an embodiment of the present invention.

[0037] Figure 3This is a schematic diagram illustrating the correspondence between the YOLO thread and the visual odometry thread in an embodiment of the present invention.

[0038] Figure 4 This is a design framework diagram of an improved visual odometer in an embodiment of the present invention.

[0039] Figure 5 These are human body extraction images in this embodiment of the invention; where (a) is a human body RGB extraction image; and (b) is a human body depth extraction image.

[0040] Figure 6 The above are human body extraction histograms in this embodiment of the invention; where (a) is the human body RGB histogram; and (b) is the human body depth histogram.

[0041] Figure 7 This is a flowchart of the mask extraction algorithm based on K-means clustering in an embodiment of the present invention.

[0042] Figure 8 This is an example diagram of segmentation mask extraction in an embodiment of the present invention.

[0043] Figure 9 This is a trajectory diagram under the fr3_s_half dataset in the embodiment of the present invention; where (a) corresponds to ORB-SLAM2; and (b) corresponds to the present invention.

[0044] Figure 10 This is a trajectory diagram under the fr3_w_half dataset in the embodiment of the present invention; where (a) corresponds to ORB-SLAM2; and (b) corresponds to the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0046] This invention provides an image processing method, which includes:

[0047] S1. Obtain the RGB image sequence and depth map sequence containing the target;

[0048] S2. Process the RGB image sequence and the depth map sequence to obtain the target depth extraction map sequence;

[0049] S3. Using a mask extraction algorithm based on K-means clustering, the target is selected from the target depth extraction map sequence to obtain the target binary mask sequence.

[0050] In this embodiment, preferably, after obtaining the target binary mask sequence, the method further includes S4: removing dynamic feature points to obtain the final segmentation mask extraction map.

[0051] In this embodiment, preferably, processing the RGB image sequence and the depth map sequence in S2 to obtain the target depth extraction map sequence includes: using a target detection model to detect targets in the RGB image sequence and obtaining detection boxes; segmenting the targets in the detection boxes from the depth map sequence based on the detection boxes to obtain the target depth extraction map sequence.

[0052] In this embodiment, preferably, the target selection process in S3 using a mask extraction algorithm based on K-means clustering to filter targets from the target depth extraction map sequence includes:

[0053] The target depth extraction map is judged by the window size. If the window is smaller than a preset threshold, the maximum and minimum depth values ​​of the target depth extraction map are directly output. If the window is not smaller than the preset threshold, the 0 values ​​are removed, and the two-dimensional depth map is converted into one-dimensional data, where the elements in the one-dimensional data are defined as depth samples.

[0054] Perform three-class classification on one-dimensional data. Set the number of clusters in the K-means clustering algorithm to 3. Perform K-means clustering and obtain the index of the smallest centroid cluster and the index of the second smallest centroid cluster from the clustering results.

[0055] Compare the sample size of the smallest centroid cluster and the second smallest centroid cluster, define the cluster with the larger sample size as the moving cluster, and calculate the standard deviation of the moving cluster;

[0056] If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than the preset percentage of the total number of samples, then the maximum and minimum depth values ​​in the motion cluster will be output.

[0057] If the standard deviation of the motion cluster is greater than the empirical value or the number of samples in the motion cluster is less than the preset percentage of the total number of samples, then K-means clustering is performed again with the number of clusters set to 2. The resulting minimum centroid cluster is defined as the motion cluster, and the maximum and minimum depth values ​​in the minimum centroid cluster are output.

[0058] In this embodiment, preferably, the K-means clustering process in S3 includes: selecting the number of clusters; initializing cluster centers; assigning data points to the nearest cluster center; updating cluster centers; repeating the assignment and update; and determining the final clustering result.

[0059] In this embodiment, preferably, after obtaining the target binary mask sequence, removing dynamic feature points in S4 includes: generating a mask pyramid based on the image pyramid; marking feature points within the mask range according to the mask pyramid, and removing dynamic feature points by preventing the feature points from generating map points.

[0060] This invention also provides an image processing apparatus, which includes:

[0061] An image sequence acquisition module is configured to acquire an RGB image sequence and a depth map sequence containing the target;

[0062] The target depth map extraction module is configured to process the RGB image sequence and the depth map sequence to obtain a target depth extraction map sequence;

[0063] The target segmentation mask extraction module is configured to use a mask extraction algorithm based on K-means clustering to filter out targets from the target depth extraction map sequence and obtain a target binary mask sequence.

[0064] In this embodiment, preferably, it further includes a dynamic feature point removal module, which is configured to generate a mask pyramid based on the image pyramid after obtaining the target binary mask sequence; mark the feature points within the mask range according to the mask pyramid, and remove the dynamic feature points by prohibiting the generation of map points by the feature points, so as to obtain the final segmentation mask extraction map.

[0065] In this embodiment, preferably, the target segmentation mask extraction module uses a K-means clustering-based mask extraction algorithm to filter targets from the target depth extraction map sequence, including:

[0066] The target depth extraction map is judged by the window size. If the window is smaller than a preset threshold, the maximum and minimum depth values ​​of the target depth extraction map are directly output. If the window is not smaller than the preset threshold, the 0 values ​​are removed, and the two-dimensional depth map is converted into one-dimensional data, where the elements in the one-dimensional data are defined as depth samples.

[0067] Perform three-class classification on one-dimensional data. Set the number of clusters in the K-means clustering algorithm to 3. Perform K-means clustering and obtain the index of the smallest centroid cluster and the index of the second smallest centroid cluster from the clustering results.

[0068] Compare the sample size of the smallest centroid cluster and the second smallest centroid cluster, define the cluster with the larger sample size as the moving cluster, and calculate the standard deviation of the moving cluster;

[0069] If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than the preset percentage of the total number of samples, then the maximum and minimum depth values ​​in the motion cluster will be output.

[0070] If the standard deviation of the motion cluster is greater than the empirical value or the number of samples in the motion cluster is less than the preset percentage of the total number of samples, then K-means clustering is performed again with the number of clusters set to 2. The resulting minimum centroid cluster is defined as the motion cluster, and the maximum and minimum depth values ​​in the minimum centroid cluster are output.

[0071] This invention also proposes a robot visual odometry method, which uses the image processing method described in the above embodiments to process image sequences captured by a camera on the robot to determine the robot's position and orientation. Specific Implementation Example 1

[0073] This invention proposes a robot visual odometry method, such as... Figure 1 As shown, this invention mainly addresses the design problem of visual odometry for robots in indoor scenarios with human movement. First, it deploys the YOLOv5 object detection algorithm, manually selecting different specific versions of YOLOv5 to prepare for subsequent work. Then, it designs a YOLOv5-based object detection program to achieve human recognition in a dynamic indoor environment. Next, it designs an improved visual odometry framework, extracting the human body's RGB image and depth map, and proposes the basic idea of ​​obtaining a two-dimensional mask for the human image. Then, it proposes a human mask extraction algorithm based on K-means clustering, performs K-means clustering, and finally achieves human mask extraction in a dynamic environment. Finally, comparative experiments verify the superior performance of the algorithms proposed in this invention.

[0074] 1) Deploy the YOLOv5 object detection algorithm

[0075] By adding a YOLOv5 object detection network to the front-end visual odometry, the RGB image of the sensor information is input, the image is converted into the network's input size, and the output is the detection bounding box, detection category, and detection confidence score detected on the RGB image. To ensure high system consistency and enable rapid inference, this invention implements the YOLOv5 model in C++. To deploy the YOLOv5 model into the system, this invention adopts the following steps:

[0076] Step 1: The trained model needs to be converted from the PT model to the ONNX model;

[0077] Step 2: Convert the ONNX model to a .trtmodel model file using TensorRT, and perform the conversion with FP32 precision;

[0078] Step 3: To adapt to the computing power and real-time requirements of different devices, the YOLOv5 model supports different variant forms, including: n, s, m, l, x. These five letters correspond to five different variants, with the computing power requirements of the devices increasing in that order and the accuracy increasing in that order.

[0079] 2) YOLOv5-based target detection program integrated into SLAM system

[0080] The model derivation process is incorporated into the ORB-SLAM2 system, and the program flow is as follows: Figure 2 As shown. In the ORB-SLAM2 system, a new thread is added to perform object detection and generate a human binary mask. This thread will continuously wait for new image input. After receiving a new image, the RGB image is converted to grayscale, and the image size is adjusted to the input size of the model. After obtaining the detection results, all detection boxes in the results are traversed, and a Detection object is generated for each detection box. The constructor of Detection constrains the original detection box range to the image range and marks whether the detection box intersects with the image boundary. If the detection box is a human, a segmentation threshold is calculated for it, and the corresponding human binary mask is generated.

[0081] To enable real-time execution of the object detection algorithm, this invention proposes an optimization scheme. During the inference process of the YOLO thread, the front-end visual odometry uses the detection results of the previous frame from the YOLO thread. The timing diagram between the two threads is as follows: Figure 3 As shown, the optimized solution does not affect the positioning accuracy. At frame i+3, both the YOLO thread and the visual odometry thread begin processing that frame. By frame i+4, the YOLO thread has not yet completed its derivation for frame i+3. Therefore, the human depth mask used by the visual odometry is generated from frame i. This processing method can greatly utilize the advantages of multi-threading, thereby improving both operational efficiency and dynamic positioning accuracy.

[0082] 3) Improved visual odometry frame

[0083] This invention provides an improved front-end visual odometer for dynamic scenes. The program framework diagram is as follows: Figure 4 As shown.

[0084] 31) Acquiring Human Body RGB Image and Depth Map: The sensor uses an RGB-D camera. Its RGB image is input into the YOLOv5 detection thread. Objects present in the current image are obtained through forward inference. Detection boxes and object labels are output to the visual odometry system. The visual odometry system filters out the object detection boxes corresponding to "human body" based on the object label information. Based on the detection boxes, the portion within the detection boxes is segmented from the original depth map to obtain the depth map corresponding to the human body. The RGB image and depth map obtained after detection box segmentation are shown below. Figure 5 As shown, a semantic segmentation network is used to extract the target contour in an RGB image.

[0085] 32) Obtain the 2D mask of the human body image: Figure 6 This is a histogram corresponding to an RGB image and a depth map. It is difficult to see the pixel distribution of the human body in the image from the RGB histogram, but the depth histogram shows that the pixel intensity is roughly divided into two parts: 40-57 and 58-80. These two parts correspond to the body depth range and background depth range in the depth map. This invention designs an algorithm for calculating the depth segmentation threshold, namely a mask extraction algorithm based on K-means clustering. This algorithm directly obtains the lower and upper limits of the human body depth in the depth map, and filters out the human body part from the depth map. The segmentation is based on the consistency of the human body depth values ​​in the depth map.

[0086] To obtain the depth segmentation threshold of the human body from the depth map of an object, this process is actually a clustering process of the object's depth map. If the foreground, midground, and background in the depth map can be obtained through a clustering algorithm, then the corresponding human body depth segmentation threshold can be obtained from them. This invention uses machine learning K-means clustering, which is the algorithm with the lowest complexity among clustering algorithms and meets the real-time requirements of front-end visual odometry. The specific implementation steps based on the K-means clustering algorithm are as follows:

[0087] Step 1: Select the number of clusters (K);

[0088] Step 2: Initialize cluster centers (Centroids);

[0089] Step 3: Assign data points to the nearest cluster centers;

[0090] Step 4: Update cluster centers;

[0091] Step 5: Repeat Step 3 and Step 4;

[0092] Step 6: Determine the final clustering results.

[0093] Based on K-means clustering, the extracted depth map can be compiled into a sample set, which is then input into the K-means clustering algorithm. The depth segmentation threshold for the human body is then calculated based on the clustering results. The specific flowchart is shown below. Figure 7 As shown. The program takes an object depth map as input and outputs the depth range of the object within the depth map, i.e., the minimum and maximum depth values.

[0094] When the program receives the depth map of an object, it first checks the window size. If the window is too small, it doesn't process it and directly returns the maximum value of the object's depth map as Max and the minimum value as Min. If the window is not smaller than the set value, it continues to the next step, removing 0 values ​​caused by camera depth anomalies, converting the two-dimensional depth map into one-dimensional data, and calling the elements in the one-dimensional data depth samples.

[0095] Since images generally contain foreground, midground, and background, this invention first performs three-class classification on the data, setting the number of clusters in the K-means clustering algorithm to 3. From the clustering results, the indices of the smallest centroid cluster and the second smallest centroid cluster are obtained. This is because a moving person will be in the foreground or midground of the camera's field of view. By comparing the sample size of these two clusters, the cluster with the larger sample size is selected and defined as the motion cluster. The standard deviation of the motion cluster is calculated. The standard deviation can reflect the dispersion of depth samples in the motion cluster. Depth samples generated by the human body generally have strong consistency. Through multiple experiments, this invention sets the empirical value of the standard deviation to 0.25.

[0096] If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than 30% of the total number of samples, then the motion cluster is considered to have strong consistency. The maximum value in the motion cluster is returned as Max, and the minimum value is returned as Min. If the standard deviation of a motion cluster is greater than the empirical value, or the number of samples in the motion cluster is less than 30% of the total number of samples, then it means that there is no mid-range scene in the scene. K-means clustering is performed again, with the number of clusters set to 2. The smallest centroid cluster obtained is defined as the motion cluster, and the maximum value in the motion cluster is returned as Max, and the minimum value is returned as Min.

[0097] After obtaining the binary mask of the human body in the image, since the system performs feature point recognition on the image pyramid, it is necessary to generate the same mask pyramid. Based on the mask pyramid, the feature points within the mask range are marked. Dynamic feature points are removed by preventing the generation of map points for these feature points, and these feature points are masked in the backend optimization.

[0098] The technical effects of the present invention were further verified through experiments.

[0099] Experiment 1 - Human Body Segmentation Mask Verification: Figure 8The images show the results of segmentation mask extraction for dynamic human bodies in the fr3_w_xyz sequence of the TUM dataset. The first row of images is the original image of the dataset, the second row is the depth map of the human body detection box, and the third row is the extracted binary mask image. The extraction results show that this invention can correctly extract the binary mask image of the human body regardless of whether there are multiple human bodies, the human body is blurred, or the human body is not fully observed.

[0100] Experiment 2 - Time Optimization Verification: A multi-threaded optimization method was designed. The specific execution time comparison is shown in Table 1. Before optimization, the average processing time for each frame of the entire system was mostly around 55ms, which was exactly 16ms longer than the ORB-SLAM2 system's average of 38ms. This 16ms was the time YOLO spent on inference. After optimization, the average time was reduced to 34ms, which was not only much faster than the sequential execution before optimization, but also faster than the ORB-SLAM2 system. This result is because, in dynamic scenes, this invention masks dynamic feature points, thus eliminating the additional computational overhead caused by dynamic feature points.

[0101] Table 1. Comparison of the time consumption (ms) of this invention and ORB-SLAM2 on the walking sequence of fr3.

[0102]

[0103] Experiment 3 - Visual Odometry Functionality Verification in Dynamic Scenes: Experiments were conducted on dynamic scenes within the publicly available TUM dataset to verify the effectiveness of the algorithm presented in this invention. A comparative analysis was performed with current leading dynamic visual odometry removal SLAM algorithms to demonstrate the superiority of the algorithm. Trajectories calculated by the ORB-SLAM2 system and the improved system of this invention were displayed in a 3D coordinate system, such as... Figure 9 and Figure 10 As shown in the figure, the actual trajectory of the camera is represented by the black dashed line, and the trajectory estimated by the system is represented by the blue solid line. Figure 9 The figure in the image is sitting in a chair with only minor movements. In the three-dimensional trajectories of these two datasets, it can be seen that the overall error of the ORB-SLAM2 system is not too large, and the estimated trajectory is roughly consistent with the real trajectory. However, some errors are relatively large. After adding dynamic feature removal, the pose estimated by this invention is more consistent with the real trajectory. Figure 10 The character in the video is constantly moving, which greatly increases the number of dynamic feature points observed by the camera. As can be seen from the pose estimated by ORB-SLAM2, there is a huge discrepancy in the estimation accuracy between the two datasets, and the entire system will no longer be stable. However, after adding dynamic feature removal, the pose calculated by this invention is very close to the real trajectory.

[0104] In these two scenario experiments, the present invention provides a three-dimensional display of the poses calculated by the two systems. The three-dimensional diagrams allow for a clear understanding that after adding dynamic feature removal, the accuracy of the system's positioning is no longer affected by the dynamic figures. The more dynamic feature points there are, the more significant the improvement of the present invention's system becomes.

[0105] The above verifications demonstrate that the present invention has relatively better performance in human body segmentation mask, real-time algorithm operation, and visual odometry.

[0106] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the RGB image sequence and depth map sequence containing the target; The RGB image sequence and the depth map sequence are processed to obtain a target depth extraction map sequence; A mask extraction algorithm based on K-means clustering is used to filter targets from the target depth extraction map sequence to obtain a target binary mask sequence; including: The target depth extraction map is judged by the window size. If the window is smaller than a preset threshold, the maximum and minimum depth values ​​of the target depth extraction map are directly output. If the window is not smaller than the preset threshold, the 0 values ​​are removed, and the two-dimensional depth map is converted into one-dimensional data, where the elements in the one-dimensional data are defined as depth samples. Perform three-class classification on one-dimensional data. Set the number of clusters in the K-means clustering algorithm to 3. Perform K-means clustering and obtain the index of the smallest centroid cluster and the index of the second smallest centroid cluster from the clustering results. Compare the sample size of the smallest centroid cluster and the second smallest centroid cluster, define the cluster with the larger sample size as the moving cluster, and calculate the standard deviation of the moving cluster; If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than the preset percentage of the total number of samples, then the maximum and minimum depth values ​​in the motion cluster will be output. If the standard deviation of the motion cluster is greater than the empirical value or the number of samples in the motion cluster is less than the preset percentage of the total number of samples, then K-means clustering is performed again with the number of clusters set to 2. The resulting minimum centroid cluster is defined as the motion cluster, and the maximum and minimum depth values ​​in the minimum centroid cluster are output.

2. The image processing method according to claim 1, characterized in that, After obtaining the target binary mask sequence, dynamic feature points are removed to obtain the final segmentation mask extraction map.

3. The image processing method according to claim 1, characterized in that, Processing the RGB image sequence and the depth map sequence to obtain a target depth extraction map sequence includes: using a target detection model to detect targets in the RGB image sequence and obtaining detection boxes; segmenting the targets in the detection boxes from the depth map sequence based on the detection boxes to obtain the target depth extraction map sequence.

4. The image processing method according to claim 2, characterized in that, After obtaining the target binary mask sequence, the removal of dynamic feature points includes: generating a mask pyramid based on the image pyramid; marking feature points within the mask range according to the mask pyramid, and removing dynamic feature points by preventing the generation of map points for these feature points.

5. An image processing apparatus, characterized in that, include: An image sequence acquisition module is configured to acquire an RGB image sequence and a depth map sequence containing the target; The target depth map extraction module is configured to process the RGB image sequence and the depth map sequence to obtain a target depth extraction map sequence; A target segmentation mask extraction module is configured to use a K-means clustering-based mask extraction algorithm to filter targets from the target depth extraction map sequence and obtain a target binary mask sequence; including: The target depth extraction map is judged by the window size. If the window is smaller than a preset threshold, the maximum and minimum depth values ​​of the target depth extraction map are directly output. If the window is not smaller than the preset threshold, the 0 values ​​are removed, and the two-dimensional depth map is converted into one-dimensional data, where the elements in the one-dimensional data are defined as depth samples. Perform three-class classification on one-dimensional data. Set the number of clusters in the K-means clustering algorithm to 3. Perform K-means clustering and obtain the index of the smallest centroid cluster and the index of the second smallest centroid cluster from the clustering results. Compare the sample size of the smallest centroid cluster and the second smallest centroid cluster, define the cluster with the larger sample size as the moving cluster, and calculate the standard deviation of the moving cluster; If the standard deviation of a motion cluster is less than the empirical value, and the number of samples in the motion cluster is greater than the preset percentage of the total number of samples, then the maximum and minimum depth values ​​in the motion cluster will be output. If the standard deviation of the motion cluster is greater than the empirical value or the number of samples in the motion cluster is less than the preset percentage of the total number of samples, then K-means clustering is performed again with the number of clusters set to 2. The resulting minimum centroid cluster is defined as the motion cluster, and the maximum and minimum depth values ​​in the minimum centroid cluster are output.

6. The image processing apparatus according to claim 5, characterized in that, It also includes a dynamic feature point removal module, which is configured to generate a mask pyramid based on the image pyramid after obtaining the target binary mask sequence; according to the mask pyramid, feature points within the mask range are marked, and dynamic feature points are removed by preventing the generation of map points by the feature points, so as to obtain the final segmentation mask extraction map.

7. A robot visual odometry system, characterized in that, The image processing method according to any one of claims 1-4 is used to process the image sequence captured by the camera on the robot to determine the position and orientation of the robot.

8. A robot visual odometry system according to claim 7, characterized in that, In the target detection thread that uses the target detection model to detect targets in the RGB image sequence, the front-end visual odometry uses the detection results of the target detection thread from the previous frame.

Images