Dynamic target identification method and system for intelligent system

Abstract

The invention provides a dynamic target identification method and system for an intelligent system. According to the method, the optical flow vector of each frame of monocular camera RGB image is analyzed through improved K-Means clustering, and a plurality of feature point sets are extracted. The method comprises the following steps: firstly, calculating a dynamic flow residual error and an epipolar matching error, obtaining a residual error mean value of each category, and determining a dynamic level through weighted matching design; for a high-dynamic category, directly deleting and marking the region of the high-dynamic category to construct a mask; and extracting new feature points in the static region. And for low dynamic points, nonlinear weighting is carried out by using a residual factor with a matching weight, and the low dynamic points and the static points are transmitted to a rear end for robust processing. According to the method, feature point sets of different spatial distributions and motion states are effectively reflected in a high-dynamic scene. The robustness of the algorithm in a complex dynamic scene is enhanced by introducing the dynamic level, so that the positioning precision is improved, and the algorithm is excellent in both low-dynamic and high-dynamic scenes.

Description

Technical Field

[0001] The present invention belongs to the field of monocular visual inertial technology, and in particular relates to a dynamic target recognition method and system for an intelligent system. Background Art

[0002] Simultaneous localization and mapping (SLAM) refers to the technology that a mobile robot equipped with sensors can map the environment and estimate its own posture in an unknown environment based on the interaction data between the sensor and the environment. At present, SLAM has been widely used in the fields of robotics, autonomous driving, augmented reality, drones, and smart homes. Visual SLAM solutions that use cameras as sensors, such as monocular cameras, binocular cameras, RGB-D cameras, etc., are widely promoted due to their low cost, low weight, low power consumption, and rich data information. Among them, the monocular SLAM solution is simple and has advantages in cost, weight, and power consumption, so it has been widely used. However, monocular SLAM still has many challenges, such as scale estimation and initialization problems. These problems can be well solved by using more complex sensors, namely binocular cameras, RGB-D cameras, etc. In addition, monocular visual inertial SLAM, adding an inertial measurement unit IMU to assist vision can also make up for the shortcomings of monocular cameras. However, these solutions are all based on the assumption of a static environment, and their performance in a dynamic environment is not satisfactory, because dynamic points violate the basic three-dimensional geometric visual constraints, thereby providing incorrect feature associations, resulting in a decrease in the accuracy of pose estimation or even failure. Some SLAM methods have added outlier removal mechanisms, such as random sampling consistency RANSAC, which can eliminate some outliers in low-dynamic environments, but this method cannot cope well if dynamic objects dominate the scene. With the rapid development of deep learning in recent years, more and more researchers use semantic information to process dynamic objects, that is, use semantic segmentation or target detection to preprocess images and remove dynamic objects in images. Semantic segmentation can provide pixel-level object masks and extract feature points outside the moving object mask. However, semantic segmentation cannot accurately determine the segmentation boundary and cannot completely cover dynamic objects. Target detection can identify dynamic objects in the form of target boxes, but it will also eliminate some static points, resulting in insufficient constraints between features. At the same time, both methods detect dynamic objects based on predefined semantic information, so they cannot identify undefined objects, such as moving tables, fluttering leaves, etc. On the other hand, semantic information cannot reflect the dynamic level, and the performance in low-dynamic scenes is poor. If the object identified as a dynamic object is actually stationary, such as a stopped car or a stationary person, segmenting it as a completely dynamic object will also reduce the accuracy of pose estimation. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a dynamic target recognition method and system for an intelligent system.

[0004] Different from the coarse-to-fine method, the present invention integrates front-end feature extraction and tracking with back-end optimization, uses a feature point clustering method based on optical flow field, dynamically segments the feature points of a new frame image before each optimization, and combines dynamic optical flow residual and epipolar matching error to calculate the weight factor reflecting the dynamic level of the feature points for back-end robust BA. The improved K-Means method of the present invention uses an adaptive weighting method to introduce feature selection into clustering analysis, thereby improving the stability and flexibility of clustering.

[0005] The technical solution of the method of the present invention is a dynamic target recognition method for an intelligent system, comprising the following steps:

[0006] Step 1: construct multiple feature points of each frame of the RGB image of the monocular camera, construct the optical flow vector of the RGB images of two adjacent frames of the monocular camera, and perform cluster analysis on the optical flow vector of each frame of the RGB image of the monocular camera through the improved K-Means clustering method to obtain multiple feature point sets of each frame of the RGB image of the monocular camera;

[0007] Step 2: Calculate the dynamic flow residual and epipolar matching error for all feature points to obtain the corresponding residual mean of each category, use the nonlinear weighted method to perform matching design, normalize the residual factor, use GNN to build the matching function, and obtain the matching weight corresponding to the residual factor output by the matching function. Perform weighted matching design to obtain the dynamic level of each category;

[0008] Step 3: For high-dynamic categories, delete them directly, mark the area where the category is located and construct a mask; use the mask to extract new feature points in the static area; for low-dynamic points, use the residual factor with matching weights for nonlinear weighting, and pass it to the backend together with the static points for robustness.

[0009] Preferably, the step 1 constructs multiple feature points of the RGB image of each frame of the monocular camera, specifically as follows:

[0010] The Harris corner point detection is used to extract multiple Harris corner points of each frame of the RGB image of each frame of the monocular camera as multiple feature points of the RGB image of each frame of the monocular camera, including calculating the image gradient, calculating the structural similarity according to the gradient, calculating the Harris response function according to the structural similarity matrix of the pixel points, and judging the corner points according to the response function value;

[0011] Step 1 constructs the optical flow vector of the RGB images of two adjacent frames of the monocular camera as follows:

[0012] Track the RGB image of each frame of the monocular camera using the Lucas-Kanade optical flow tracking method to obtain the pixel coordinates of each feature point of the RGB image of the current frame of the monocular camera and the pixel coordinates of each feature point of the RGB image of the next frame of the monocular camera;

[0013] Construct an optical flow vector of the RGB images of two adjacent monocular camera frames according to the pixel coordinates of each feature point of the RGB image of the current frame of the monocular camera and the pixel coordinates of each feature point of the RGB image of the next frame of the monocular camera;

[0014] Preferably, in step 1, cluster analysis is performed on the optical flow vector of each frame of the RGB image of the monocular camera, as follows:

[0015] Each feature point corresponds to an optical flow vector. Each frame of the image has multiple feature points, that is, multiple optical flow vectors. Multiple optical flow vectors form an optical flow field. Multiple optical flow vectors are clustered to distinguish different feature points according to the optical flow characteristics.

[0016] In step 1, the optical flow vector of each frame of the RGB image of the monocular camera is clustered and analyzed by the improved K-Means clustering method, as follows:

[0017] Step 1.1: Calculate the feature weight of each feature point corresponding to each optical flow vector;

[0018] The feature weight of each feature point contained in the optical flow vector of each frame of the monocular camera's RGB image;

[0019] Each optical flow vector of the RGB image of each frame of the monocular camera is taken as input, and the output is the feature weight of each optical flow amount.

[0020] The loss function is defined by minimizing the distance from each optical flow vector of the RGB image of each frame of the monocular camera to the corresponding cluster center and maximizing the distance between the optical flow vectors of different cluster centers. The loss function converges by continuously iteratively adjusting the weights. The specific formula is as follows:

[0021]

[0022] Among them, ClassDistLoss represents the distance loss, InterDist i Represents the distance from the i-th optical flow vector to the corresponding cluster center, Corr i,,i It is the correlation between each feature point corresponding to each optical flow vector of the RGB image of each frame of the monocular camera.

[0023] Step 1.2: Adaptive feature weighting;

[0024] The adaptive feature weighting is specifically as follows:

[0025] When calculating the distance between each optical flow vector and the cluster center of each frame of the RGB image of the monocular camera, the weighted Euclidean distance of the optical flow vector is obtained by weighted summation of each optical flow vector:

[0026] Define the inter-class distance between each optical flow vector:

[0027]

[0028] Among them, InterDist i represents the distance from the optical flow vector i to the center of its corresponding cluster, x i represents the amount of light loss i, Represents the cluster center of the cluster where the light loss i is located.

[0029] Define the correlation between features:

[0030]

[0031] Among them, Corr j,k is the correlation between feature points j and k represents the optical flow value of the cluster center of the i-th cluster; x j is the optical flow vector of the jth feature point, x k is the optical flow vector of the kth feature point

[0032] Finally, combining the inter-class distance and feature correlation, the inter-class distance loss function is defined:

[0033]

[0034] Among them, Corr i,j is the correlation between all feature points i and j

[0035] In each clustering iteration, the feature weights are dynamically adjusted in the direction of the negative gradient of the loss function, and steps 1.1 and 1.2 are repeated until the inter-class distance loss function is less than the threshold to adapt to the feature distribution of different clusters.

[0036] The technical solution of the system of the present invention is a dynamic target recognition system for intelligent systems, including:

[0037] A feature point set construction module is used to construct multiple feature points of each frame of the RGB image of the monocular camera, construct the optical flow vector of the RGB images of two adjacent frames of the monocular camera, and perform cluster analysis on the optical flow vector of each frame of the RGB image of the monocular camera through the improved K-Means clustering method to obtain multiple feature point sets of each frame of the RGB image of the monocular camera;

[0038] The dynamic level matching calculation module is used to calculate the dynamic flow residual and polar line matching error for all feature points, obtain the corresponding residual mean of each category, use the nonlinear weighted method to perform matching design, normalize the residual factor, use GNN to build the matching function, and the matching function outputs the residual factor to obtain the matching weight, perform weighted matching design, and obtain the dynamic level of each category;

[0039] The mask construction module is used to directly delete high-dynamic categories, mark the area where the category is located and construct a mask; use the mask to extract new feature points in the static area; for low-dynamic points, use the residual factor with matching weights for nonlinear weighting, and pass it to the backend together with the static points for robustness.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] Improved K-Means clustering analysis based on optical flow, the clustering results are more stable and flexible, especially in high-dynamic scenes, and the results can reflect the feature point sets with different spatial distributions and motion states;

[0042] The introduction of dynamic levels can improve the robustness of the algorithm in various complex dynamic scenes, and can greatly improve positioning accuracy regardless of low-dynamic scenes or high-dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 : A flow chart of a method according to an embodiment of the present invention.

[0044] Figure 2 : Schematic diagram of the implementation scenario of an embodiment of the present invention.

[0045] Figure 3 : Optical flow clustering result diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0048] The following is combined with Figure 1-3 The specific implementation of the present invention is a dynamic target recognition method and system for intelligent systems, which are as follows:

[0049] like Figure 1 Shown is a method flow chart of an embodiment of the present invention.

[0050] like Figure 1 Shown is a diagram of an implementation scenario of an embodiment of the present invention.

[0051] The left picture shows a positioning terminal equipped with the proposed algorithm, and the right picture shows an experimental scene where data is collected in an open playground. Two people make irregular movements in front of the camera, acting as dynamic objects to test the positioning accuracy of the positioning terminal.

[0052] The specific implementation of the method of the present invention is a dynamic target recognition method for an intelligent system, which is as follows:

[0053] like Figure 1 FIG. 1 is a flow chart of a method according to an embodiment of the present invention.

[0054] Step 1: Extract multiple Harris corner points from each frame of the RGB image of each frame of the monocular camera through Harris corner detection as multiple feature points of the RGB image of each frame of the monocular camera, including calculating the image gradient, and calculating the structural similarity based on the gradient, and then calculating the Harris response function based on the structural similarity matrix of the pixel points, and finally judging which pixels are corner points based on the response function value.

[0055] Track the RGB image of each frame of the monocular camera using the Lucas-Kanade optical flow tracking method to obtain the pixel coordinates of each feature point of the RGB image of the current frame of the monocular camera and the pixel coordinates of each feature point of the RGB image of the next frame of the monocular camera;

[0056] Construct an optical flow vector of the RGB images of two adjacent monocular camera frames according to the pixel coordinates of each feature point of the RGB image of the current frame of the monocular camera and the pixel coordinates of each feature point of the RGB image of the next frame of the monocular camera;

[0057] The optical flow vector of each frame of the RGB image of the monocular camera is clustered and analyzed by the improved K-Means clustering method to obtain multiple feature point sets of each frame of the RGB image of the monocular camera;

[0058] The cluster analysis of the optical flow vector of each frame of the RGB image of the monocular camera is as follows:

[0059] Each feature point corresponds to an optical flow vector. Each frame of the image has multiple feature points, that is, multiple optical flow vectors. Multiple optical flow vectors form an optical flow field. Multiple optical flow vectors are clustered to distinguish different feature points according to the optical flow characteristics.

[0060] In step 1, the optical flow vector of each frame of the RGB image of the monocular camera is clustered and analyzed by the improved K-Means clustering method, as follows:

[0061] Step 1.1: Calculate the feature weight of each feature point corresponding to each optical flow vector;

[0062] The feature weight of each feature point contained in the optical flow vector of each frame of the monocular camera's RGB image;

[0063] Each optical flow vector of the RGB image of each frame of the monocular camera is taken as input, and the output is the feature weight of each optical flow amount.

[0064] The loss function is defined by minimizing the distance from each optical flow vector of the RGB image of each frame of the monocular camera to the corresponding cluster center and maximizing the distance between the optical flow vectors of different cluster centers. The loss function converges by continuously iteratively adjusting the weights. The specific formula is as follows:

[0065]

[0066] Among them, ClassDistLoss represents the distance loss, InterDist i Represents the distance from the i-th optical flow vector to the corresponding cluster center, Corr i,,i It is the correlation between each feature point corresponding to each optical flow vector of the RGB image of each frame of the monocular camera.

[0067] Step 1.2: Adaptive feature weighting;

[0068] The adaptive feature weighting is specifically as follows:

[0069] When calculating the distance between each optical flow vector and the cluster center of each frame of the RGB image of the monocular camera, the weighted Euclidean distance of the optical flow vector is obtained by weighted summation of each optical flow vector:

[0070] Define the inter-class distance between each optical flow vector:

[0071]

[0072] Among them, InterDist i represents the distance from the optical flow vector i to the center of its corresponding cluster, x i represents the amount of light loss i, Represents the cluster center of the cluster where the light loss i is located.

[0073] Define the correlation between features:

[0074]

[0075] Among them, Corr j,k is the correlation between feature points j and k represents the optical flow value of the cluster center of the i-th cluster; x j is the optical flow vector of the jth feature point, x k is the optical flow vector of the kth feature point

[0076] Finally, combining the inter-class distance and feature correlation, the inter-class distance loss function is defined:

[0077]

[0078] Among them, Corr i,j is the correlation between all feature points i and j

[0079] In each clustering iteration, the feature weights are dynamically adjusted in the direction of the negative gradient of the loss function, and steps 1.1 and 1.2 are repeated until the inter-class distance loss function is less than the threshold to adapt to the feature distribution of different clusters.

[0080] like Figure 3 As shown, optical flow vectors of different colors represent different clustering results. It can be seen that the optical flow vectors on dynamic objects are obviously different from those on static objects.

[0081] Step 2: Calculate the dynamic flow residual and polar line matching error for all feature points, and then obtain the corresponding residual mean for each category. Use the nonlinear weighted method for matching design. First, normalize the residual factor, and then use GNN to build a matching function. The matching function outputs the residual factor to obtain the matching weight, and then perform weighted matching design to finally obtain the dynamic level of each category.

[0082] Step 3: For high-dynamic categories, delete them directly and mark the area where the category is located to make a mask. Then use the mask to extract new feature points in the static area. For low-dynamic points, use the residual factor with matching weights for nonlinear weighting and pass it to the backend together with the static points for robust BA.

[0083] The specific implementation of the system of the present invention is a dynamic target recognition system for intelligent systems, including:

[0084] A feature point set construction module is used to construct multiple feature points of each frame of the RGB image of the monocular camera, construct the optical flow vector of the RGB images of two adjacent frames of the monocular camera, and perform cluster analysis on the optical flow vector of each frame of the RGB image of the monocular camera through the improved K-Means clustering method to obtain multiple feature point sets of each frame of the RGB image of the monocular camera;

[0085] The dynamic level matching calculation module is used to calculate the dynamic flow residual and polar line matching error for all feature points, obtain the corresponding residual mean of each category, use the nonlinear weighted method to perform matching design, normalize the residual factor, use GNN to build the matching function, and the matching function outputs the residual factor to obtain the matching weight, perform weighted matching design, and obtain the dynamic level of each category;

[0086] The mask construction module is used to directly delete the high-dynamic category, mark the area where the category is located and construct a mask; use the mask to extract new feature points in the static area; for low-dynamic points, use the residual factor with matching weights for nonlinear weighting, and pass it to the backend together with the static points for robustness, as shown in Table 1.

[0087] Table 1: Test results of four dynamic scenes

[0088]

[0089] The above are the test results of the proposed method in four dynamic scenes. The RMSE (m) of the absolute trajectory error is at the centimeter level.

[0090] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0091] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A dynamic target recognition method for intelligent systems, characterized in that: The following steps are involved: Step 1: construct multiple feature points of each frame of the RGB image of the monocular camera, construct the optical flow vector of the RGB images of two adjacent frames of the monocular camera, and perform cluster analysis on the optical flow vector of each frame of the RGB image of the monocular camera through the improved K-Means clustering method to obtain multiple feature point sets of each frame of the RGB image of the monocular camera; Step 2: Calculate the dynamic flow residual and epipolar matching error for all feature points to obtain the corresponding residual mean of each category, use the nonlinear weighted method to perform matching design, normalize the residual factor, use GNN to build the matching function, and obtain the matching weight corresponding to the residual factor output by the matching function. Perform weighted matching design to obtain the dynamic level of each category; Step 3: For high-dynamic categories, delete them directly, mark the area where the category is located and construct a mask; use the mask to extract new feature points in the static area; for low-dynamic points, use the residual factor with matching weights for nonlinear weighting, and pass it to the backend together with the static points for robustness.

2. The method for dynamic target recognition for intelligent systems according to claim 1, characterized in that: Step 1 constructs multiple feature points of the RGB image of each frame of the monocular camera, as follows: Harris corner detection is used to extract multiple Harris corner points from each frame of the RGB image of each frame of the monocular camera as multiple feature points of the RGB image of each frame of the monocular camera, including calculating the image gradient, calculating the structural similarity based on the gradient, calculating the Harris response function based on the structural similarity matrix of the pixel points, and judging the corner points based on the response function value.

3. The method for dynamic target recognition for intelligent systems according to claim 2, characterized in that: Step 1 constructs the optical flow vector of the RGB images of two adjacent frames of the monocular camera as follows: Track the RGB image of each frame of the monocular camera using the Lucas-Kanade optical flow tracking method to obtain the pixel coordinates of each feature point of the RGB image of the current frame of the monocular camera and the pixel coordinates of each feature point of the RGB image of the next frame of the monocular camera; According to the pixel coordinates of each feature point of the RGB image of the current frame of the monocular camera and the pixel coordinates of each feature point of the RGB image of the next frame of the monocular camera, the optical flow vectors of the RGB images of two adjacent frames of the monocular camera are constructed.

4. The method for dynamic target recognition for intelligent systems according to claim 3, characterized in that: Step 1 performs cluster analysis on the optical flow vector of each frame of the RGB image of the monocular camera, as follows: Each feature point corresponds to an optical flow vector. Each frame image has multiple feature points, that is, multiple optical flow vectors. Multiple optical flow vectors form an optical flow field. Multiple optical flow vectors are clustered to distinguish different feature points according to the optical flow characteristics.

5. The method for dynamic target recognition for intelligent systems according to claim 4, characterized in that: In step 1, the optical flow vector of each frame of the RGB image of the monocular camera is clustered and analyzed by the improved K-Means clustering method, as follows: Step 1.1: Calculate the feature weight of each feature point corresponding to each optical flow vector; Step 1.2: Adaptive feature weighting.

6. The method for dynamic target recognition for intelligent systems according to claim 5, characterized in that: The feature weights of the feature points corresponding to each optical flow vector are calculated as described in step 1.1, as follows: The feature weight of each feature point contained in the optical flow vector of each frame of the monocular camera's RGB image; Take each optical flow vector of the RGB image of each frame of the monocular camera as input, and output the feature weight of each optical flow amount; The loss function is defined by minimizing the distance from each optical flow vector of the RGB image of each frame of the monocular camera to the corresponding cluster center and maximizing the distance between the optical flow vectors of different cluster centers. The loss function converges by continuously iteratively adjusting the weights. The specific formula is as follows: Among them, ClassDistLoss represents the distance loss, InterDist i Represents the distance from the i-th optical flow vector to the corresponding cluster center, Corr i,,i It is the correlation between each feature point corresponding to each optical flow vector of the RGB image of each frame of the monocular camera.

7. The method for dynamic target recognition for intelligent systems according to claim 6, characterized in that: The adaptive feature weighting described in step 1.2 is as follows: The adaptive feature weighting is specifically as follows: When calculating the distance between each optical flow vector and the cluster center of each frame of the RGB image of the monocular camera, the weighted Euclidean distance of the optical flow vector is obtained by weighted summation of each optical flow vector: Define the inter-class distance between each optical flow vector: Among them, InterDist i represents the distance from the optical flow vector i to the center of its corresponding cluster, x i represents the amount of light loss i, Represents the cluster center of the cluster where the light loss i is located.

8. The method for dynamic target recognition for intelligent systems according to claim 7, characterized in that: Define the correlation between features: Among them, Corr j,k is the correlation between feature points j and k represents the optical flow value of the cluster center of the i-th cluster; x j is the optical flow vector of the jth feature point, x k is the optical flow vector of the kth feature point. Finally, combining the inter-class distance and feature correlation, the inter-class distance loss function is defined: Among them, Corr i,j is the correlation between all feature points i and j.

9. The method for dynamic target recognition for intelligent systems according to claim 8, characterized in that: In each clustering iteration, the feature weights are dynamically adjusted in the direction of the negative gradient of the loss function, and steps 1.1 and 1.2 are repeated until the inter-class distance loss function is less than the threshold to adapt to the feature distribution of different clusters.

10. A dynamic target recognition system for intelligent systems, characterized in that: include: A feature point set construction module is used to construct multiple feature points of each frame of the RGB image of the monocular camera, construct the optical flow vector of the RGB images of two adjacent frames of the monocular camera, and perform cluster analysis on the optical flow vector of each frame of the RGB image of the monocular camera through the improved K-Means clustering method to obtain multiple feature point sets of each frame of the RGB image of the monocular camera; The dynamic level matching calculation module is used to calculate the dynamic flow residual and polar line matching error for all feature points, obtain the corresponding residual mean of each category, use the nonlinear weighted method to perform matching design, normalize the residual factor, use GNN to build the matching function, and the matching function outputs the residual factor to obtain the matching weight, perform weighted matching design, and obtain the dynamic level of each category; The mask construction module is used to directly delete the highly dynamic category, mark the area where the category is located and construct a mask; use the mask to extract new feature points in the static area; For low dynamic points, nonlinear weighting is performed using residual factors with matching weights and passed to the backend together with static points for robustness.

Images