Mobile robot dynamic obstacle real-time detection method and system based on embedded vision

Abstract

The invention relates to a mobile robot dynamic obstacle real-time detection method and system based on embedded vision. The method comprises the following steps: performing motion compensation on a historical frame image, generating a motion compensation image, performing frame difference analysis on the motion compensation image and a current frame image, performing adaptive threshold segmentation in combination with local texture features, and generating a motion candidate mask; and extracting feature points in the mask region, calculating an optical flow vector and generating RGB image blocks, and performing dynamic attribute verification after fusion to obtain a verified dynamic point set. And further calculating an optical flow direction angle and a fusion speed value, constructing a motion feature vector and clustering, and generating a central position coordinate of the obstacle instance. And through rotation matrix conversion and Kalman filtering processing, a smooth position estimation value and a smooth speed estimation value are obtained, and the relative distance and the dynamic threat degree of the obstacle are calculated. According to the method, through motion compensation, filtering optimization and the like, the real-time performance, precision and motion estimation accuracy of dynamic obstacle detection in an embedded scene are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a method and system for real-time detection of dynamic obstacles in mobile robots based on embedded vision. Background Technology

[0002] With the rapid development of industrial automation, service robots and autonomous driving, mobile robots have higher requirements for real-time detection of dynamic obstacles. Embedded vision, due to its low cost, high integration and rapid response, has become the core technology path for environmental perception of mobile robots.

[0003] Currently, dynamic obstacle detection methods based on embedded vision mainly include frame difference methods, optical flow methods, and deep learning-based target detection methods. However, existing dynamic obstacle detection methods still have many shortcomings in practical applications. For example, traditional frame difference methods do not compensate for the robot's own motion state, which easily leads to a large number of false motion regions due to robot movement. Furthermore, when using fixed threshold segmentation, it is difficult to adapt to the local texture feature differences of frame difference maps in different scenarios, resulting in limited detection accuracy. Although the full-image optical flow method can capture pixel motion information, it requires extracting feature points from the entire image and calculating optical flow, which consumes a lot of computing power and cannot meet the real-time requirements of embedded platforms. Moreover, some neural network-based methods rely only on single image appearance features for dynamic attribute judgment without integrating motion information such as optical flow, resulting in insufficient accuracy of dynamic target verification. In addition, the network structure is complex and difficult to deploy efficiently on embedded devices. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and system for real-time dynamic obstacle detection of mobile robots based on embedded vision to address the above-mentioned technical problems. This aims to improve the detection accuracy, motion estimation accuracy, and threat assessment practicality, and meet the safe operation requirements of mobile robots in complex scenarios.

[0005] In a first aspect, this application provides a real-time dynamic obstacle detection method for mobile robots based on embedded vision, including:

[0006] Motion compensation is performed on historical frame images based on robot odometry data to obtain motion-compensated images. Frame difference analysis is performed on the current frame image and the motion-compensated image to generate a frame difference map. Adaptive threshold segmentation is performed by combining the local texture features of the frame difference map to generate motion candidate masks.

[0007] Feature points are extracted within the region defined by the motion candidate mask, the optical flow vector of the feature points is calculated, and RGB image patches corresponding to the feature points are generated by combining the current frame image; the RGB image patches and optical flow vectors are fused to generate fused data blocks, and the fused data blocks are input into a preset neural network for dynamic attribute verification to generate dynamic probability values ​​and velocity scalar values. Thresholding processing is performed based on the dynamic probability values ​​to obtain the verified dynamic point set;

[0008] For each point in the verified dynamic point set, the corresponding optical flow direction angle is calculated; based on the optical flow vector and velocity scalar value, a fused velocity value is generated; combined with the position coordinates and optical flow direction angle of each point in the verified dynamic point set, a motion feature vector is constructed; based on the motion feature vector, motion consistency clustering is performed to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance.

[0009] The robot coordinate system velocity components are constructed based on the dominant direction angle and average velocity value. A rotation matrix is ​​constructed based on the robot heading angle. Coordinate transformation is performed on the robot coordinate system velocity components to generate global velocity components. Kalman filtering is then performed on the center position coordinates to generate smooth position estimates and smooth velocity estimates.

[0010] The Euclidean distance between the smoothed position estimate and the position coordinates in the current robot pose is calculated to generate the relative distance to the obstacle; the threat level is calculated based on the smoothed velocity estimate and the relative distance to the obstacle to generate the dynamic threat level value.

[0011] In one embodiment, motion compensation is performed on historical frame images based on robot odometry data to obtain motion-compensated images. Frame difference analysis is then performed on the current frame image and the motion-compensated image to generate a frame difference map. Adaptive threshold segmentation is then performed using the local texture features of the frame difference map to generate motion candidate masks, including:

[0012] A transformation matrix is ​​generated by constructing a transformation matrix based on the translation components and rotation angles in the robot odometry data.

[0013] Image transformation processing is performed on historical frame images based on affine transformation matrix to generate motion-compensated images;

[0014] Perform pixel-level grayscale difference calculation on the current frame image and the motion-compensated image to generate an absolute frame difference map, and use the absolute frame difference map as the frame difference map.

[0015] The frame difference map is processed by 3×3 neighborhood gradient variance calculation to generate local texture complexity value, and the local texture complexity value is used as local texture feature.

[0016] Based on the comparison of local texture features with the thresholds for flat regions and texture-rich regions, a preset low threshold is selected when the local texture features are less than the threshold for flat regions, and a preset high threshold is selected when the local texture features are greater than the threshold for texture-rich regions.

[0017] The frame difference map is binarized and segmented based on the selected preset low threshold or preset high threshold to generate motion candidate masks.

[0018] In one embodiment, feature points are extracted within the region defined by the motion candidate mask, the optical flow vector of the feature points is calculated, and RGB image patches corresponding to the feature points are generated by combining them with the current frame image; the RGB image patches and the optical flow vectors are fused to generate fused data blocks, and the fused data blocks are input into a preset neural network for dynamic attribute verification, generating dynamic probability values ​​and velocity scalar values. Thresholding processing is performed based on the dynamic probability values ​​to obtain a verified dynamic point set, including:

[0019] In the pixel region with a value of 1 in the motion candidate mask, Shi-Tomasi corner detection is performed to extract feature points and generate a feature point set;

[0020] For each feature point in the feature point set, perform Lucas-Kanade optical flow iterative calculation based on historical frame images and the current frame image to generate an optical flow vector;

[0021] Using the position coordinates of each feature point as the center, the current frame image is cropped to a fixed size region to generate RGB image blocks;

[0022] The optical flow vector is normalized to generate normalized optical flow data;

[0023] The normalized optical flow data is used as two additional channels and merged with the RGB image block to generate a fused data block.

[0024] The fused data block is input into a preset neural network for forward inference processing to generate dynamic probability values ​​and velocity scalar values.

[0025] The dynamic probability values ​​are subjected to threshold filtering, and feature points with dynamic probability values ​​greater than the preset probability threshold are retained to generate a dynamic point set after verification.

[0026] In one embodiment, for each point in the verified dynamic point set, the corresponding optical flow direction angle is calculated; a fused velocity value is generated based on the optical flow vector and velocity scalar value; and a motion feature vector is constructed by combining the position coordinates and optical flow direction angle of each point in the verified dynamic point set; motion consistency clustering is performed based on the motion feature vector to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance, including:

[0027] The horizontal and vertical components of the optical flow vector at each point in the verified dynamic point set are calculated using arctangent to generate the optical flow direction angle.

[0028] The magnitude of the optical flow vector at each point in the verified dynamic point set is multiplied with the corresponding velocity scalar value to generate a fused velocity value.

[0029] The position coordinates, corresponding optical flow direction angles, and fusion velocity values ​​of each point in the verified dynamic point set are combined to generate a motion feature vector.

[0030] For each point in the verified dynamic point set, a preset distance function is used to perform DBSCAN clustering based on the corresponding motion feature vector to generate a set of clusters. The distance function is a weighted sum of spatial Euclidean distance and motion feature distance.

[0031] The mean value of the position coordinates of each point within each cluster in the cluster set is calculated to generate the center position coordinates of the obstacle instance;

[0032] The median of the fusion velocity values ​​of each point within each cluster in the cluster set is calculated to generate the average velocity value of the obstacle instance;

[0033] The optical flow direction angle of each point within each cluster in the cluster set is calculated by circular averaging to generate the dominant direction angle of the obstacle instance.

[0034] In one embodiment, robot coordinate system velocity components are constructed based on the dominant direction angle and average velocity value, a rotation matrix is ​​constructed based on the robot heading angle, coordinate transformation is performed on the robot coordinate system velocity components to generate global velocity components, and Kalman filtering is performed on the center position coordinates to generate smooth position estimates and smooth velocity estimates, including:

[0035] Vector decomposition is performed based on the dominant direction angle and average velocity value to generate horizontal and vertical velocity components in the robot coordinate system, thus obtaining the robot coordinate system velocity components.

[0036] Based on the robot's heading angle, construct a rotation matrix with the cosine, negative sine, sine, and cosine values ​​of the robot's heading angle as elements;

[0037] The robot coordinate system velocity components are multiplied with the rotation matrix to generate global velocity components in the global coordinate system.

[0038] The position state corresponding to the center position coordinates and the velocity state corresponding to the global velocity components are used as the state vector of the Kalman filter. The center position coordinates and global velocity components are used as observations and input into the Kalman filter for state estimation. Through filtering iteration, smooth position estimates and smooth velocity estimates are obtained.

[0039] In one embodiment, the preset neural network includes a compressed network, a dynamic attribute branch, and a velocity regression branch. The compressed network is obtained by compressing the number of channels of each convolutional layer of the backbone network using MobileNetV3-Small as the backbone network.

[0040] The fused data block is input into a pre-defined neural network for forward inference processing to generate dynamic probability values ​​and velocity scalar values, including:

[0041] The fused data blocks are processed by a compression network to extract features and generate multi-scale feature maps.

[0042] After performing global average pooling on the multi-scale feature map through dynamic attribute branching, classification is performed based on the Sigmoid activation layer to generate dynamic probability values.

[0043] After performing global average pooling on the multi-scale feature maps based on the velocity regression branch, a linear output layer is used for regression processing to generate velocity scalar values.

[0044] In one embodiment, the dynamic threat level is calculated using the following formula:

[0045]

[0046] in, This is a dynamic threat level value. This is the distance weighting coefficient. For speed weighting coefficients, The relative distance to the obstacle. Distance threshold This represents the global velocity vector of the obstacle, i.e., the smoothed velocity estimate. The angle between the direction of the obstacle's velocity and the direction of the line connecting the robot and the obstacle. , Let the position vector of the robot be the position vector from the obstacle. This is the speed threshold.

[0047] Secondly, this application also provides a real-time dynamic obstacle detection system for mobile robots based on embedded vision, including:

[0048] The motion compensation and mask generation module is used to perform motion compensation on historical frame images based on robot odometry data to obtain motion-compensated images, perform frame difference analysis on the current frame image and the motion-compensated image to generate a frame difference map, and perform adaptive threshold segmentation based on the local texture features of the frame difference map to generate motion candidate masks.

[0049] The dynamic point verification module is used to extract feature points within the region defined by the motion candidate mask, calculate the optical flow vector of the feature points, and generate RGB image blocks corresponding to the feature points by combining them with the current frame image; it fuses the RGB image blocks and optical flow vectors to generate fused data blocks, and inputs the fused data blocks into a preset neural network for dynamic attribute verification, generating dynamic probability values ​​and velocity scalar values, and performing threshold filtering based on the dynamic probability values ​​to obtain the verified dynamic point set;

[0050] The motion feature clustering module is used to calculate the corresponding optical flow direction angle for each point in the verified dynamic point set; generate a fused velocity value based on the optical flow vector and velocity scalar value; and construct a motion feature vector by combining the position coordinates and optical flow direction angle of each point in the verified dynamic point set; and perform motion consistency clustering based on the motion feature vector to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance.

[0051] The global motion estimation and filtering module is used to construct the robot coordinate system velocity components based on the dominant direction angle and average velocity value, construct the rotation matrix based on the robot heading angle, perform coordinate transformation on the robot coordinate system velocity components to generate global velocity components, and perform Kalman filtering processing in combination with the center position coordinates to generate smooth position estimates and smooth velocity estimates.

[0052] The threat calculation module is used to calculate the Euclidean distance between the smoothed position estimate and the position coordinates in the current robot pose to generate the relative distance to the obstacle; and to calculate the threat level based on the smoothed velocity estimate and the relative distance to the obstacle to generate a dynamic threat level value.

[0053] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the first aspect.

[0054] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the first aspect.

[0055] The aforementioned method and system for real-time dynamic obstacle detection in mobile robots based on embedded vision first performs motion compensation and frame difference analysis using robot odometry data. Combined with local texture features, adaptive threshold segmentation is used to generate motion candidate masks, avoiding false regions caused by uncompensated robot motion in traditional frame difference methods. Second, feature points are extracted from the mask-defined region, and optical flow vectors are calculated. RGB image patches and optical flow vectors are fused to generate fused data blocks, and their dynamic properties are verified. This not only reduces the computational cost of full-image optical flow calculation but also compensates for the insufficient accuracy of single-feature verification. Furthermore, motion feature vectors are constructed based on position, optical flow direction angle, and fusion velocity. Obstacle instance parameters are generated through motion consistency clustering, overcoming the lack of instantiated motion estimation in existing methods and improving the completeness of obstacle motion state description. Finally, state estimation is optimized through coordinate transformation and Kalman filtering. The dynamic threat level is quantified by combining relative distance and velocity, effectively improving the stability of motion estimation and providing a decision-making basis for subsequent robot path planning. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart of a real-time dynamic obstacle detection method for mobile robots based on embedded vision, provided as an exemplary embodiment of the present invention;

[0058] Figure 2 A flowchart illustrating a method for obtaining a verified dynamic point set, as provided in an exemplary embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of a real-time dynamic obstacle detection system for mobile robots based on embedded vision, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] In one embodiment, such as Figure 1As shown, a real-time dynamic obstacle detection method for mobile robots based on embedded vision is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0062] S101: Motion compensation is performed on historical frame images based on robot odometry data to obtain motion-compensated images. Frame difference analysis is performed on the current frame image and the motion-compensated image to generate a frame difference map. Adaptive threshold segmentation is performed by combining the local texture features of the frame difference map to generate motion candidate masks.

[0063] Specifically, during operation, the translational and rotational movements of a mobile robot cause global offsets between adjacent frames. Directly calculating frame differences can misinterpret pixel changes caused by the robot's motion as dynamic obstacles, resulting in numerous false detections. Therefore, motion compensation processing can be performed on historical frame images using motion data from the robot's odometry. By constructing a transformation relationship that matches the robot's motion, the offsets caused by robot movement in historical frame images are offset, generating a motion-compensated image aligned with the current frame scene. This ensures that the difference between the two frames is caused only by external dynamic obstacles. Subsequently, frame difference analysis can be performed on the current frame and the motion-compensated image to generate a frame difference map, which highlights the areas of change between the two frames, areas that are likely to contain dynamic obstacles. Furthermore, considering the differences in local texture features of frame difference maps under different scenarios, fixed threshold segmentation may lead to missed detections or false detections in some areas. Therefore, adaptive threshold segmentation can be performed by combining local texture features of frame difference maps. The segmentation threshold is dynamically adjusted according to the texture complexity of each pixel's neighborhood. A lower threshold is used for flat areas with simple textures to avoid missed detections, while a higher threshold is used for areas with complex textures to reduce noise interference. Finally, motion candidate masks that can accurately mark potential dynamic regions are generated.

[0064] S102: Extract feature points within the region defined by the motion candidate mask, calculate the optical flow vector of the feature points, and generate RGB image blocks corresponding to the feature points by combining them with the current frame image; fuse the RGB image blocks and the optical flow vector to generate fused data blocks, and input the fused data blocks into a preset neural network for dynamic attribute verification, generate dynamic probability values ​​and velocity scalar values, perform threshold filtering based on the dynamic probability values, and obtain the verified dynamic point set.

[0065] Specifically, due to the limited computing resources of embedded platforms, performing feature point extraction and optical flow calculation on the entire image would result in a large amount of redundant computation, failing to meet real-time detection requirements. Therefore, feature points can be extracted within the potential dynamic region marked by motion candidate masks, significantly narrowing the scope of feature point extraction and reducing computational consumption. The extracted feature points can be further processed using optical flow calculations to obtain their motion information. By tracking the positional changes of feature points between historical frames and the current frame, an optical flow vector representing the motion trend of the feature points is generated. Furthermore, RGB image patches corresponding to the feature points can be generated by combining the current frame image. These RGB image patches contain the appearance information of the feature points. By fusing the RGB image patches with the optical flow vectors, both appearance and motion features are integrated, resulting in a fused data block that more comprehensively describes the characteristics of the feature points. Subsequently, the fused data block is input into a preset neural network for dynamic attribute verification. This neural network, by learning from a large amount of sample data, can automatically extract features from the fused data block and determine whether it belongs to a dynamic obstacle, thereby generating dynamic probability values ​​and velocity scalar values. The dynamic probability value represents the probability that the feature point belongs to a dynamic obstacle, while the velocity scalar value estimates the speed of the dynamic obstacle. Threshold filtering based on dynamic probability values ​​can filter out feature points with dynamic probability values ​​higher than a set threshold, resulting in a verified dynamic point set. This effectively removes background noise and interference from other non-dynamic objects, improving the accuracy of dynamic obstacle detection.

[0066] S103: For each point in the verified dynamic point set, calculate the corresponding optical flow direction angle; generate a fused velocity value based on the optical flow vector and velocity scalar value; construct a motion feature vector by combining the position coordinates and optical flow direction angle of each point in the verified dynamic point set; perform motion consistency clustering based on the motion feature vector to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance.

[0067] Specifically, each point in the verified dynamic point set can only reflect the motion information of a single pixel, and cannot directly represent the complete obstacle entity. Furthermore, dynamic points corresponding to different obstacles may be mixed together. Therefore, clustering is required to instantiate the obstacles. Illustratively, for each point in the verified dynamic point set, its corresponding optical flow direction angle is calculated. This optical flow direction angle provides directional information about the dynamic obstacle's motion. Subsequently, based on the optical flow vector and velocity scalar value, a fused velocity value can be generated. This value integrates velocity magnitude and direction information, and can more accurately describe the motion state of the dynamic obstacle. By combining the position coordinates and optical flow direction angle of each point in the verified dynamic point set, a motion feature vector is constructed. This motion feature vector contains information such as the position, direction, and velocity of the dynamic point, and can comprehensively describe the motion characteristics of the dynamic obstacle. Then, motion consistency clustering is performed based on the motion feature vector, clustering dynamic points with similar motion characteristics into the same obstacle instance. For example, during the clustering process, the similarity between motion feature vectors, such as differences in velocity direction and magnitude, can be analyzed to divide the dynamic points into different clusters. Each cluster represents a dynamic obstacle instance, thereby generating the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance.

[0068] S104: Construct the robot coordinate system velocity components based on the dominant direction angle and average velocity value, construct the rotation matrix based on the robot heading angle, perform coordinate transformation on the robot coordinate system velocity components to generate global velocity components, and perform Kalman filtering processing in combination with the center position coordinates to generate smooth position estimates and smooth velocity estimates.

[0069] Specifically, the average velocity and dominant direction angle of the obstacle obtained in step S103 are defined based on the robot coordinate system. However, robot path planning needs to be performed in the global coordinate system, thus requiring coordinate system transformation. By decomposing the velocity of the dynamic obstacle into components of the robot coordinate system, the direction and magnitude of the obstacle's motion can be understood more intuitively. A rotation matrix is ​​constructed based on the robot's heading angle, and coordinate transformation is performed on the velocity components of the robot coordinate system to generate global velocity components. These global velocity components can represent the motion state of the dynamic obstacle in the global coordinate system. Furthermore, due to the unavoidable noise interference during image detection and sensor measurement, the directly obtained center position coordinates and global velocity components will fluctuate, affecting the accuracy of motion state estimation. Therefore, Kalman filtering can be applied to the center position coordinates to generate smooth position and velocity estimates. Kalman filtering is a recursive filtering method that can smoothly estimate the position and velocity of the obstacle based on its historical motion information and current observation data, thereby effectively reducing measurement noise and errors and improving the accuracy and smoothness of dynamic obstacle motion estimation.

[0070] S105: Calculate the Euclidean distance between the smoothed position estimate and the position coordinates in the current robot pose to generate the relative distance to the obstacle; calculate the threat level based on the smoothed velocity estimate and the relative distance to the obstacle to generate the dynamic threat level value.

[0071] Specifically, simply obtaining the position and velocity information of obstacles is insufficient for the robot to determine the hazard priority of different obstacles; therefore, it is necessary to quantify the threat level through threat calculation. For example, the Euclidean distance between the smoothed position estimate and the robot's current pose coordinates can be calculated to obtain the relative distance to the obstacle. This distance directly reflects the proximity of the obstacle to the robot; the closer the distance, the higher the potential hazard. Simultaneously, the modulus of the smoothed velocity estimate can be calculated to obtain the actual scalar value of the obstacle's motion velocity. The higher the velocity, the more pronounced the obstacle's tendency to approach or move away from the robot. Finally, based on the relative distance and velocity scalar value, the threat level can be calculated using the following formula:

[0072]

[0073] in, This is a dynamic threat level value. This is the distance weighting coefficient. For speed weighting coefficients, The relative distance to the obstacle. Distance threshold This represents the global velocity vector of the obstacle, i.e., the smoothed velocity estimate. The angle between the direction of the obstacle's velocity and the direction of the line connecting the robot and the obstacle. , Let the position vector of the robot be the position vector from the obstacle. This is the speed threshold.

[0074] The above formula, through a weighted fusion of two terms, can comprehensively consider the dynamic interaction between distance and velocity, where the distance term... One part reflects the characteristic that the closer the target, the greater the danger; the other part reflects the characteristic that the greater the speed and the more the direction is directed towards the robot, the greater the danger. Furthermore, this formula achieves a smooth non-linear mapping by introducing an exponential function, which avoids drastic fluctuations in threat level due to small changes in distance or speed, resulting in a more stable output. In addition, a distance weighting coefficient is used... The weights of distance and speed factors can be flexibly adjusted to adapt to obstacle avoidance requirements in different scenarios. The final generated dynamic threat level value... It can transform the position and motion state of obstacles into intuitive decision indicators, enabling the robot path planning module to quickly identify high-risk obstacles and formulate safe and efficient obstacle avoidance strategies.

[0075] The aforementioned method first generates motion-compensated images by performing motion compensation on historical frame images. Then, it combines frame difference analysis and adaptive threshold segmentation to generate motion candidate masks, improving the accuracy of dynamic obstacle detection. Secondly, feature points are extracted within the mask region, and optical flow vectors are calculated. RGB image patches are fused with optical flow information for dynamic attribute verification, overcoming the limitations of single-feature detection methods and improving the accuracy of dynamic target verification. Furthermore, motion feature vectors are constructed using optical flow direction angles and fused velocity values, and motion consistency clustering is performed to generate the center position, average velocity, and dominant direction of obstacle instances, effectively addressing the problems of insufficient dynamic obstacle instantiation and feature extraction. Finally, coordinate transformation and Kalman filtering are used to generate smooth position and velocity estimates, and dynamic threat levels are calculated, solving the real-time problems of dynamic obstacle tracking and threat assessment, and significantly improving the robot's dynamic obstacle detection and avoidance performance in complex environments.

[0076] In one embodiment, motion compensation is performed on historical frame images based on robot odometry data to obtain motion-compensated images. Frame difference analysis is then performed on the current frame image and the motion-compensated image to generate a frame difference map. Adaptive threshold segmentation is then performed using the local texture features of the frame difference map to generate motion candidate masks, including:

[0077] A transformation matrix is ​​generated by constructing a transformation matrix based on the translation components and rotation angles in the robot odometry data.

[0078] Image transformation processing is performed on historical frame images based on affine transformation matrix to generate motion-compensated images; pixel-level grayscale difference processing is performed on current frame images and motion-compensated images to generate absolute frame difference maps, which are then used as frame difference maps.

[0079] The frame difference map is processed by 3×3 neighborhood gradient variance calculation to generate local texture complexity value, and the local texture complexity value is used as local texture feature.

[0080] Based on the comparison of local texture features with the thresholds for flat regions and texture-rich regions, a preset low threshold is selected when the local texture features are less than the threshold for flat regions, and a preset high threshold is selected when the local texture features are greater than the threshold for texture-rich regions.

[0081] The frame difference map is binarized and segmented based on the selected preset low threshold or preset high threshold to generate motion candidate masks.

[0082] Specifically, we can start by using the translation component output by the robot's odometry. and rotation angle Constructing the affine transformation matrix This matrix is ​​used to simulate the effect of the robot's motion on historical frame images, and its mathematical expression can be:

[0083]

[0084] This matrix can achieve rotation transformation through trigonometric function terms and position offset through translation terms, which can accurately describe the spatial transformation relationship of historical frame images caused by robot movement, providing a mathematical basis for subsequent motion compensation.

[0085] Based on the aforementioned affine transformation matrix, pixel resampling algorithms such as bilinear interpolation can be used to map the coordinates of each pixel in the historical frame image, transforming the pixels in the historical frame image according to the transformation matrix. The rules are mapped to new coordinate positions, and interpolation is performed to fill the pixel gaps that appear after the transformation, resulting in a motion compensation map aligned with the static background of the current frame image. After motion compensation, the current frame image can be... With motion-compensated images Perform pixel-level grayscale difference calculations to generate an absolute frame difference map. The calculation method can be as follows:

[0086]

[0087] Specifically, absolute frame difference map Pixel regions with high grayscale values ​​can be considered potential dynamic obstacle regions. However, due to factors such as image texture and noise, direct segmentation using a fixed threshold can lead to missed or false detections in some areas. Therefore, it is necessary to combine the local texture features of the frame difference map for adaptive threshold adjustment. For example, a gradient operator such as the Sobel operator can be used to calculate the gradient matrix in the 3×3 neighborhood of each pixel, and then the variance of the gradient matrix can be calculated to obtain the local texture complexity of the pixel. Among them, flat areas, such as solid-color walls, have low texture complexity and small gradient variance, while textured areas, such as vegetation and textured ground, have higher texture complexity and larger gradient variance. To illustrate, a threshold for flat areas can be preset. And texture rich region threshold Local texture complexity In cases where the flat area has a simple texture, a small difference in grayscale can be identified as a dynamic obstacle, and a preset low threshold can be selected. To avoid missed detections. When the local texture complexity When the grayscale difference is large in areas with rich texture, a preset high threshold can be selected. This filters out false detections caused by texture noise. Finally, the absolute frame difference map is binarized and segmented according to a selected preset low or high threshold to generate motion candidate masks. The motion candidate mask can be defined as:

[0088]

[0089] The regions with a value of 1 in the aforementioned motion candidate mask are the selected potential dynamic obstacle regions. This mask significantly narrows the scope of subsequent feature extraction and optical flow calculation, reducing the computational power consumption of the embedded platform and improving the accuracy of dynamic region detection through an adaptive threshold strategy.

[0090] In one embodiment, such as Figure 2 As shown, the verified dynamic point set can be obtained through the following steps.

[0091] S201: Perform Shi-Tomasi corner detection processing within the pixel region with a value of 1 in the motion candidate mask, extract feature points, and generate a feature point set;

[0092] S202: For each feature point in the feature point set, perform Lucas-Kanade optical flow iterative calculation based on historical frame images and the current frame image to generate an optical flow vector;

[0093] S203: Using the position coordinates of each feature point as the center, perform fixed-size region cropping on the current frame image to generate RGB image blocks;

[0094] S204: Normalize the optical flow vector to generate normalized optical flow data. Use the normalized optical flow data as two additional channels and merge them with the RGB image block to generate a fused data block.

[0095] S205: Input the fused data block into the preset neural network for forward inference processing to generate dynamic probability values ​​and velocity scalar values. Then, perform threshold filtering on the dynamic probability values, retain feature points with dynamic probability values ​​greater than the preset probability threshold, and generate a verified dynamic point set.

[0096] Specifically, Shi-Tomasi corner detection can be performed first within pixel regions with a value of 1 in the motion candidate mask. By calculating the eigenvalues ​​of the local autocorrelation matrix of the image, points with significant gray-level changes in multiple directions can be selected as feature points. These feature points exhibit good inter-frame tracking stability. By generating a feature point set containing all candidate dynamic feature points, a clear tracking target can be provided for subsequent optical flow calculations. Furthermore, Lucas-Kanade iterative optical flow calculations can be performed on each feature point in the feature point set to obtain its motion information. This calculation is based on the assumption of constant pixel brightness. For each feature point, its neighborhood window can be selected in the historical frame image. By minimizing the pixel brightness error between the corresponding window in the current frame and the historical frame, an optical flow equation set can be established and iteratively solved to finally obtain the optical flow vector characterizing the motion trend of the feature point. ,in , These represent the pixel-level motion components in the horizontal and vertical directions, respectively. This optical flow vector visually reflects the direction and velocity of feature points' motion between frames.

[0097] Since judging dynamic attributes solely through optical flow vectors has limitations—for example, noise points in static areas may produce false optical flow due to image texture—complementary verification can be performed by combining the appearance information of feature points. For instance, using the position coordinates of each feature point as the center, the current frame image is cropped to a fixed size, such as 32×32 pixels, generating an RGB image patch containing texture and color information surrounding the feature point. This image patch provides the appearance dimension features of the feature point, helping to distinguish between dynamic objects and static noise. Furthermore, the optical flow vector can be normalized, mapping its numerical range to the [-1,1] interval, eliminating scale differences in optical flow amplitude across different scenes, and ensuring unified processing of optical flow features and appearance features in subsequent neural networks. Subsequently, the normalized optical flow data (containing horizontal and vertical channels) is merged with the RGB image patch (containing R, G, and B channels) to generate a five-channel fused data block, achieving multimodal fusion of motion and appearance features, laying the data foundation for the neural network to simultaneously learn the complementary information of the two features.

[0098] Specifically, the fused data block is input into a pre-defined lightweight neural network for forward inference. This network, designed for embedded platforms, employs efficient structures such as depthwise separable convolutions to output dynamic probability values ​​and velocity scalar values. The dynamic probability values ​​range from [0,1], with values ​​closer to 1 indicating higher confidence in the dynamic attribute. The velocity scalar value reflects the speed of the feature point's movement. Subsequently, the dynamic probability values ​​are filtered based on a pre-defined probability threshold, such as 0.8, retaining feature points with dynamic probability values ​​higher than the threshold, ultimately generating a validated dynamic point set. This process effectively filters out static noise points and low-confidence false positives, ensuring that the point set processed subsequently consists of high-confidence dynamic feature points, providing reliable input for obstacle instance clustering and motion state estimation.

[0099] In one embodiment, the preset neural network includes a compressed network, a dynamic attribute branch, and a velocity regression branch. The compressed network is obtained by compressing the number of channels in each convolutional layer of the backbone network using MobileNetV3-Small as the backbone network.

[0100] The fused data block is input into a pre-defined neural network for forward inference processing to generate dynamic probability values ​​and velocity scalar values, including:

[0101] The fused data blocks are processed by a compression network to extract features and generate multi-scale feature maps.

[0102] After performing global average pooling on the multi-scale feature map through the dynamic attribute branch, classification is performed based on the Sigmoid activation layer to generate dynamic probability values; after performing global average pooling on the multi-scale feature map through the velocity regression branch, regression is performed through the linear output layer to generate velocity scalar values.

[0103] Specifically, considering the core constraints of limited computing power and storage resources on the embedded platform of the mobile robot, conventional deep learning networks such as ResNet and the ordinary MobileNet series cannot meet the real-time requirements of dynamic obstacle detection due to their large parameter scale and redundant computation. Therefore, this embodiment uses a lightweight pre-designed neural network to minimize computational overhead while ensuring feature extraction capabilities. The core structure of this pre-designed neural network includes a compressed network, a dynamic attribute branch, and a velocity regression branch. The compressed network serves as the core of feature extraction and is optimized based on the MobileNetV3-Small backbone network. MobileNetV3-Small itself, through a combination of depthwise separable convolutions, channel attention mechanisms, and nonlinear activation functions, already possesses the advantages of low computational cost and high feature representation, making it suitable for mobile scenarios. However, in order to further adapt to the extreme computing power of the embedded platform, this embodiment compresses the number of channels in each convolutional layer, that is, compresses the number of output channels of each convolutional layer of the original backbone network to one-quarter of the original size (e.g., the original 64 channels are compressed to 16 channels), reduces the channel dimension of the feature map, thereby reducing the total number of parameters and the amount of computation, while retaining the hierarchical structure and attention mechanism of the original network, which can effectively avoid excessive loss of feature extraction capability.

[0104] This compressed network enables multi-scale feature extraction from a five-channel fused data block (3 RGB channels + 2 normalized optical flow channels). For example, the initial convolutional layer of the compressed network (16 channels after compression, 3×3 kernel size) unifies the dimensionality and performs preliminary feature encoding on the five-channel input. Subsequently, six sequentially stacked depthwise separable convolutional modules (including a BatchNorm layer and an h-swish activation function) are used. Each module employs depthwise convolution (channel-wise convolution to reduce spatial dimensionality) and pointwise convolution (1×1 convolution to adjust channel dimensionality), thereby progressively extracting multi-scale feature maps from low to high while reducing computational cost. Low-level feature maps (such as the outputs of the first three modules) retain pixel-level detail, suitable for capturing features of small obstacles. High-level feature maps (such as the outputs of the last three modules) can achieve global semantic information aggregation through receptive field expansion, suitable for feature representation of large obstacles or complex motion states. The final multi-scale feature maps are fused through channel concatenation, providing comprehensive feature input for subsequent dual-task branches.

[0105] Furthermore, the dynamic attribute branch in the pre-defined neural network is geared towards binary classification tasks to determine whether the target corresponding to a feature point is a dynamic obstacle or static noise. In this branch, global average pooling is first performed on the multi-scale fused feature map. This involves averaging all pixel values ​​in each channel of the feature map to generate a one-dimensional feature vector with dimensions equal to the number of compressed channels. This significantly reduces the parameter size to adapt to embedded platforms while aggregating global information from multi-scale features, avoiding interference from local noise in the classification results. The one-dimensional feature vector is then input into a fully connected layer (64 neurons) for feature mapping, and finally, a Sigmoid activation layer outputs a dynamic probability value. The Sigmoid function maps the output to the [0,1] interval; a value closer to 1 indicates a higher confidence level for the feature point corresponding to a dynamic obstacle, while a value closer to 0 indicates a higher confidence level for static noise, thus meeting the probability output requirements of binary classification tasks. The velocity regression branch is geared towards continuous value regression tasks, quantifying the motion velocity scalar of feature points. Its feature preprocessing can be consistent with the dynamic attribute branch, i.e., global average pooling is performed on the multi-scale fused feature map to generate a one-dimensional feature vector, ensuring consistency in feature input between the two branches. Furthermore, the regression task requires outputting continuous velocity scalar values. Therefore, after feature mapping in the fully connected layer (32 neurons), this branch can use a linear output layer instead of the activation layer. This linear output layer has no activation function constraints and can directly output continuous values ​​positively correlated with the motion velocity of feature points as velocity scalar values. These values ​​can also be calibrated with labels during network training (using manually labeled actual velocities as labels), thus accurately quantifying the actual motion velocity of feature points and compensating for the limitation of optical flow vectors, which only reflect relative motion.

[0106] In one embodiment, for each point in the verified dynamic point set, the corresponding optical flow direction angle is calculated; a fused velocity value is generated based on the optical flow vector and velocity scalar value; and a motion feature vector is constructed by combining the position coordinates and optical flow direction angle of each point in the verified dynamic point set; motion consistency clustering is performed based on the motion feature vector to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance, including:

[0107] The horizontal and vertical components of the optical flow vector at each point in the verified dynamic point set are processed by arctangent calculation to generate the optical flow direction angle; the magnitude of the optical flow vector at each point in the verified dynamic point set is multiplied by the corresponding velocity scalar value to generate the fused velocity value.

[0108] The position coordinates, corresponding optical flow direction angles, and fusion velocity values ​​of each point in the verified dynamic point set are combined to generate a motion feature vector.

[0109] For each point in the verified dynamic point set, a preset distance function is used to perform DBSCAN clustering based on the corresponding motion feature vector to generate a set of clusters. The distance function is a weighted sum of spatial Euclidean distance and motion feature distance.

[0110] The mean value of the position coordinates of each point in each cluster is calculated to generate the center position coordinates of the obstacle instance; the median value of the fusion velocity of each point in each cluster is calculated to generate the average velocity value of the obstacle instance; and the circular mean value of the optical flow direction angle of each point in each cluster is calculated to generate the dominant direction angle of the obstacle instance.

[0111] Specifically, after verification, each point in the dynamic point set can only reflect the motion and appearance features of a single pixel, and cannot directly represent the complete obstacle entity. Furthermore, dynamic points on the same obstacle surface tend to cluster due to similar spatial distribution and consistent motion states, while dynamic points on different obstacles exhibit dispersion due to differences in motion trends. Therefore, this embodiment uses feature refinement, vector construction, and clustering instantiation to transform discrete points into structured obstacle instance parameters. Illustratively, the optical flow vector of each dynamic point is first analyzed for direction, where the optical flow vector... horizontal components With vertical component Through the arctangent function Calculate the optical flow direction angle The range of values ​​for this angle is: It can accurately quantify the direction of motion of feature points.

[0112] However, the magnitude of the optical flow vector can only reflect the relative motion speed at the pixel level, and its accuracy is easily affected by image resolution and texture noise. In contrast, the speed scalar value output by the pre-defined neural network is an optimized result based on multimodal feature learning of appearance and motion, and has stronger robustness. Therefore, the two can be multiplied to generate a fused speed value. Then, by complementing and fusing pixel-level motion trends with deep learning optimization results, the real-time motion response characteristics of the optical flow vector can be preserved, and the speed accuracy can be calibrated through the network output, thereby improving the reliability of motion speed representation.

[0113] Specifically, the two-dimensional position coordinates (x, y), optical flow direction angle, and fused velocity value of the dynamic point can be combined into a four-dimensional motion feature vector. Then, the DBSCAN clustering algorithm is selected. This algorithm does not require a preset number of clusters and can adaptively identify clusters of arbitrary shapes and remove isolated noise points, making it suitable for the actual scenario of uncertain numbers and irregular shapes of dynamic obstacles in this embodiment. The key to clustering is a custom distance function. In this embodiment, the function is constructed using a weighted sum of spatial Euclidean distance and motion feature distance, and its expression can be:

[0114]

[0115] in, Represents the i-th and j-th motion feature vectors , The distance between them and and are the two-dimensional position coordinates of the i-th and j-th dynamic points, respectively, reflecting their spatial position in the image. and These are the fusion speed values ​​for the i-th and j-th dynamic points, respectively. and These are the optical flow direction angles at the i-th and j-th dynamic points, respectively. Spatial weighting coefficient, These are motion weighting coefficients; the sum of the two is 1, and they can be dynamically adjusted according to the scene.

[0116] The aforementioned distance function achieves dual constraints of spatial clustering and motion consistency through weighted fusion. This avoids mis-clustering of points with different motions but close spatial proximity due to a single spatial distance, or missed clustering of points that are spatially dispersed but have consistent motion due to a single motion distance. Ultimately, it generates a set of clusters corresponding to a complete obstacle for each cluster. After cluster generation, the parameters of points within each cluster can be statistically aggregated to extract the core parameters of the obstacle instance. The center position coordinates can be calculated using the mean of all point coordinates within the cluster, providing a reliable location basis for subsequent collision detection. The average velocity value can be calculated using the median of the fused velocity values ​​within the cluster. This median is more robust to extreme outliers, such as abnormal velocities caused by noise points, avoiding velocity estimation bias caused by outliers skewing the mean. Since angles have periodicity (e.g., 10° and 350° are practically close), ordinary arithmetic means will produce significant errors. The dominant direction angle can be calculated using a circular mean, thus accurately aggregating the motion directions of points within the cluster to obtain the dominant motion orientation of the obstacle.

[0117] In one embodiment, robot coordinate system velocity components are constructed based on the dominant direction angle and average velocity value, a rotation matrix is ​​constructed based on the robot heading angle, coordinate transformation is performed on the robot coordinate system velocity components to generate global velocity components, and Kalman filtering is performed on the center position coordinates to generate smooth position estimates and smooth velocity estimates, including:

[0118] Vector decomposition is performed based on the dominant direction angle and average velocity value to generate horizontal and vertical velocity components in the robot coordinate system, thus obtaining the robot coordinate system velocity components; based on the robot heading angle, a rotation matrix is ​​constructed with the cosine, negative sine, sine, and cosine values ​​of the robot heading angle as elements.

[0119] The robot coordinate system velocity components are multiplied with the rotation matrix to generate global velocity components in the global coordinate system.

[0120] The position state corresponding to the center position coordinates and the velocity state corresponding to the global velocity components are used as the state vector of the Kalman filter. The center position coordinates and global velocity components are used as observations and input into the Kalman filter for state estimation. Through filtering iteration, smooth position estimates and smooth velocity estimates are obtained.

[0121] Specifically, the dominant direction angle and average velocity values ​​of obstacle instances obtained from clustering are local motion parameters based on the robot's own coordinate system (with the robot's center of mass as the origin and the forward direction as the X-axis). However, the path planning of the mobile robot needs to be performed in a global coordinate system, such as the map coordinate system, and the directly output position and velocity parameters are easily affected by image noise and sensor errors, resulting in fluctuations. Therefore, this embodiment uses coordinate transformation and state filtering to achieve globalization and smoothing of the parameters. For example, based on the dominant direction angle of the obstacle instance... and average speed value Vector decomposition is performed to transform the scalar velocity into two-dimensional velocity vector components in the robot coordinate system. This dominant direction angle... Characterizes the orientation of the obstacle in the robot coordinate system, and its average velocity value. The magnitude of the motion speed can be represented by trigonometric decomposition, yielding the horizontal velocity component in the robot coordinate system. and vertical velocity components Its calculation formula can be:

[0122]

[0123] Furthermore, it can also be based on the robot's own heading angle. Construct the rotation matrix. The angle between the robot's forward direction and the X-axis of the global coordinate system reflects the attitude relationship between the robot's coordinate system and the global coordinate system. This rotation matrix can be:

[0124]

[0125] In the aforementioned matrix, trigonometric function transformations can eliminate the attitude differences between the two coordinate systems. Furthermore, the combination of cosine and sine terms enables rotational mapping of vectors in the two-dimensional plane, ensuring that the direction and magnitude of the velocity components remain physically consistent during coordinate transformation, thus avoiding misjudgments of motion state due to coordinate system differences. Subsequently, based on this matrix, by representing the velocity components in the robot coordinate system as column vectors and performing matrix multiplication with the rotation matrix, the velocity components can be transformed from the robot coordinate system to the global coordinate system, generating global velocity components. This ensures that the obstacle's motion velocity parameters are consistent with the global coordinate system of path planning, guaranteeing that the robot can accurately determine the obstacle's motion trend in the global environment and providing parameter support that meets the scenario's requirements for obstacle avoidance decisions.

[0126] Specifically, due to texture noise in the image detection process, measurement errors in odometry data, and statistical biases in the clustering process, the directly obtained obstacle center position coordinates and global velocity components will exhibit random fluctuations, affecting the stability of motion state estimation. Therefore, Kalman filtering can be used for state optimization. For example, the obstacle's position state (center position coordinates) and velocity state (global velocity components) can be combined into a four-dimensional state vector. This vector can encompass the obstacle's spatial position and velocity information, meeting the Kalman filter's requirement for estimating the position-velocity coupled state. Simultaneously, the center position coordinates and global velocity components are used as observations and input into the Kalman filter for iterative estimation. The Kalman filter can achieve state optimization through two iterative steps. In the prediction stage, based on the smoothed state estimate and state transition matrix from the previous time step, the prior state estimate for the current time step is predicted. In the update stage, the Kalman gain is calculated by combining the current observations and the observation matrix to correct the prior estimate, obtaining the posterior estimate for the current time step, i.e., the smoothed position estimate and smoothed velocity estimate. This filtering process, through a mathematical model, can suppress the influence of random noise, reduce the fluctuation of the output state parameters, and retain the true motion trend of the obstacle. It avoids response lag caused by over-filtering and can provide high-precision and high-stability motion state data for subsequent threat level calculation.

[0127] Based on the same inventive concept, such as Figure 3 As shown, this application also provides a real-time obstacle detection system 300 for mobile robots based on embedded vision, used to implement the aforementioned real-time obstacle detection method for mobile robots based on embedded vision. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the real-time obstacle detection system for mobile robots based on embedded vision provided below can be found in the limitations of the various method embodiments above, and will not be repeated here. The system includes:

[0128] The motion compensation and mask generation module 301 is used to perform motion compensation on historical frame images based on robot odometry data to obtain motion-compensated images, perform frame difference analysis on the current frame image and the motion-compensated image to generate a frame difference map, and perform adaptive threshold segmentation based on the local texture features of the frame difference map to generate motion candidate masks.

[0129] The dynamic point verification module 302 is used to extract feature points within the region defined by the motion candidate mask, calculate the optical flow vector of the feature points, and generate RGB image blocks corresponding to the feature points in combination with the current frame image; fuse the RGB image blocks and the optical flow vector to generate fused data blocks, and input the fused data blocks into a preset neural network for dynamic attribute verification, generating dynamic probability values ​​and velocity scalar values, and performing threshold filtering based on the dynamic probability values ​​to obtain the verified dynamic point set;

[0130] The motion feature clustering module 303 is used to calculate the corresponding optical flow direction angle for each point in the verified dynamic point set; generate a fused velocity value based on the optical flow vector and velocity scalar value; construct a motion feature vector by combining the position coordinates and optical flow direction angle of each point in the verified dynamic point set; and perform motion consistency clustering processing based on the motion feature vector to generate the center position coordinates, average velocity value and dominant direction angle of the obstacle instance.

[0131] The global motion estimation and filtering module 304 is used to construct the robot coordinate system velocity components based on the dominant direction angle and average velocity value, construct the rotation matrix based on the robot heading angle, perform coordinate transformation on the robot coordinate system velocity components to generate global velocity components, and perform Kalman filtering processing in combination with the center position coordinates to generate smooth position estimates and smooth velocity estimates.

[0132] The threat calculation module 305 is used to perform Euclidean distance calculation based on the smoothed position estimate and the position coordinates in the current robot pose to generate the relative distance to the obstacle; and to perform threat calculation based on the smoothed velocity estimate and the relative distance to the obstacle to generate a dynamic threat value.

[0133] In one exemplary embodiment, the present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the real-time dynamic obstacle detection method for mobile robots based on embedded vision of this application. A multi-core processor is preferred to improve the parallel processing capability of the system. The memory provides sufficient temporary storage space to support program execution and data processing. The memory capacity should be large enough to accommodate large amounts of data and computational tasks.

[0134] In one exemplary embodiment, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the mobile robot dynamic obstacle real-time detection method based on embedded vision of this application. The computer-readable storage medium may include: read-only memory, random access memory (RAM), solid-state drive (SSD), or optical disk, etc.

[0135] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for real-time dynamic obstacle detection in mobile robots based on embedded vision, characterized in that, The method includes: Motion compensation is performed on historical frame images based on robot odometry data to obtain motion-compensated images. Frame difference analysis is performed on the current frame image and the motion-compensated images to generate a frame difference map. Adaptive threshold segmentation is then performed based on the local texture features of the frame difference map to generate motion candidate masks. Feature points are extracted within the region defined by the motion candidate mask, the optical flow vector of the feature points is calculated, and an RGB image patch corresponding to the feature points is generated by combining the current frame image; the RGB image patch and the optical flow vector are fused to generate a fused data block, and the fused data block is input into a preset neural network for dynamic attribute verification to generate dynamic probability values ​​and velocity scalar values. Threshold filtering is performed based on the dynamic probability values ​​to obtain a verified dynamic point set. For each point in the verified dynamic point set, the corresponding optical flow direction angle is calculated; a fused velocity value is generated based on the optical flow vector and the velocity scalar value; a motion feature vector is constructed by combining the position coordinates of each point in the verified dynamic point set and the optical flow direction angle; motion consistency clustering is performed based on the motion feature vector to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance. Based on the dominant direction angle and the average velocity value, a robot coordinate system velocity component is constructed. Based on the robot heading angle, a rotation matrix is ​​constructed. The robot coordinate system velocity component is subjected to coordinate transformation to generate a global velocity component. Combined with the center position coordinates, Kalman filtering is performed to generate a smooth position estimate and a smooth velocity estimate. Based on the smoothed position estimate and the position coordinates in the current robot pose, Euclidean distance is calculated to generate the relative distance to the obstacle; based on the smoothed velocity estimate and the relative distance to the obstacle, threat level is calculated to generate a dynamic threat level value.

2. The method according to claim 1, characterized in that, The process involves performing motion compensation on historical frame images based on robot odometry data to obtain motion-compensated images, performing frame difference analysis on the current frame image and the motion-compensated images to generate a frame difference map, and performing adaptive threshold segmentation based on the local texture features of the frame difference map to generate motion candidate masks, including: Based on the translation components and rotation angles in the robot odometry data, a transformation matrix is ​​constructed to generate an affine transformation matrix. The historical frame image is transformed based on the affine transformation matrix to generate the motion-compensated image. Perform pixel-level grayscale difference calculation processing on the current frame image and the motion-compensated image to generate an absolute frame difference map, and use the absolute frame difference map as the frame difference map; The frame difference map is processed by 3×3 neighborhood gradient variance calculation to generate local texture complexity value, and the local texture complexity value is used as the local texture feature; The local texture features are compared with the flat region threshold and the texture-rich region threshold respectively. When the local texture features are less than the flat region threshold, a preset low threshold is selected. When the local texture features are greater than the texture-rich region threshold, a preset high threshold is selected. The frame difference map is binarized and segmented according to the selected preset low threshold or preset high threshold to generate the motion candidate mask.

3. The method according to claim 1, characterized in that, The process involves extracting feature points within the region defined by the motion candidate mask, calculating the optical flow vector of the feature points, and generating RGB image patches corresponding to the feature points by combining them with the current frame image. The RGB image patches and the optical flow vectors are then fused to generate fused data blocks. These fused data blocks are input into a preset neural network for dynamic attribute verification, generating dynamic probability values ​​and velocity scalar values. Threshold filtering is performed based on the dynamic probability values ​​to obtain a verified dynamic point set, including: In the pixel region with a value of 1 in the motion candidate mask, Shi-Tomasi corner detection processing is performed to extract the feature points and generate a feature point set; For each feature point in the feature point set, the optical flow vector is generated by performing Lucas-Kanade optical flow iterative calculation based on the historical frame image and the current frame image. Using the position coordinates of each feature point as the center, the current frame image is cropped to a fixed size region to generate the RGB image block; The optical flow vector is normalized to generate normalized optical flow data; The normalized optical flow data is used as two additional channels and merged with the RGB image block to generate the fused data block. The fused data block is input into the preset neural network for forward inference processing to generate the dynamic probability value and the velocity scalar value. The dynamic probability values ​​are subjected to threshold filtering to retain feature points whose dynamic probability values ​​are greater than a preset probability threshold, thereby generating the verified dynamic point set.

4. The method according to claim 1, characterized in that, For each point in the verified dynamic point set, the corresponding optical flow direction angle is calculated; a fused velocity value is generated based on the optical flow vector and the velocity scalar value; and a motion feature vector is constructed by combining the position coordinates of each point in the verified dynamic point set and the optical flow direction angle. Based on the motion feature vectors, motion consistency clustering is performed to generate the center position coordinates, average velocity value, and dominant direction angle of obstacle instances, including: The horizontal and vertical components of the optical flow vector at each point in the verified dynamic point set are subjected to arctangent calculation to generate the optical flow direction angle. The magnitude of the optical flow vector of each point in the verified dynamic point set is multiplied with the corresponding velocity scalar value to generate a fused velocity value. The position coordinates of each point in the verified dynamic point set, the corresponding optical flow direction angle, and the fusion velocity value are combined to generate the motion feature vector. For each point in the verified dynamic point set, a preset distance function is used to perform DBSCAN clustering based on the corresponding motion feature vector to generate a cluster set. The distance function is a weighted sum of spatial Euclidean distance and motion feature distance. The mean value of the position coordinates of each point within each cluster in the cluster set is calculated to generate the center position coordinates of the obstacle instance; The median of the fusion velocity value of each point within each cluster in the cluster set is calculated to generate the average velocity value of the obstacle instance; The optical flow direction angle of each point within each cluster in the cluster set is calculated by circular averaging to generate the dominant direction angle of the obstacle instance.

5. The method according to claim 1, characterized in that, The process involves constructing robot coordinate system velocity components based on the dominant direction angle and the average velocity value, constructing a rotation matrix based on the robot heading angle, performing coordinate transformation on the robot coordinate system velocity components to generate global velocity components, and combining these with the center position coordinates to perform Kalman filtering to generate smooth position estimates and smooth velocity estimates, including: Based on the dominant direction angle and the average velocity value, vector decomposition processing is performed to generate horizontal and vertical velocity components in the robot coordinate system, thus obtaining the robot coordinate system velocity components. Based on the robot's heading angle, construct the rotation matrix with the cosine, negative sine, sine, and cosine values ​​of the robot's heading angle as elements; The robot coordinate system velocity components are multiplied with the rotation matrix to generate the global velocity components in the global coordinate system. The position state corresponding to the center position coordinates and the velocity state corresponding to the global velocity component are used as the state vector of the Kalman filter. The center position coordinates and the global velocity component are used as observations and input into the Kalman filter for state estimation. The smoothed position estimate and the smoothed velocity estimate are obtained through filtering iteration correction.

6. The method according to claim 3, characterized in that, The preset neural network includes a compressed network, a dynamic attribute branch, and a velocity regression branch. The compressed network is obtained by compressing the number of channels in each convolutional layer of the backbone network using MobileNetV3-Small as the backbone network. The step of inputting the fused data block into the preset neural network for forward inference processing to generate the dynamic probability value and the velocity scalar value includes: The fused data block is processed by the compression network to extract features and generate a multi-scale feature map. After performing global average pooling on the multi-scale feature map through the dynamic attribute branch, classification is performed based on the Sigmoid activation layer to generate the dynamic probability value. After performing global average pooling on the multi-scale feature map based on the velocity regression branch, the velocity scalar value is generated by performing regression processing through a linear output layer.

7. The method according to claim 1, characterized in that, The dynamic threat level is calculated using the following formula: in, The dynamic threat level value, This is the distance weighting coefficient. For speed weighting coefficients, The relative distance to the obstacle. Distance threshold This represents the global velocity vector of the obstacle, i.e., the smoothed velocity estimate. The angle between the direction of the obstacle's velocity and the direction of the line connecting the robot and the obstacle. , Let be the position vector of the robot to the obstacle. This is the speed threshold.

8. A real-time dynamic obstacle detection system for mobile robots based on embedded vision, characterized in that, The system includes: The motion compensation and mask generation module is used to perform motion compensation on historical frame images based on robot odometry data to obtain motion-compensated images, perform frame difference analysis on the current frame image and the motion-compensated images to generate a frame difference map, and perform adaptive threshold segmentation based on the local texture features of the frame difference map to generate motion candidate masks. The dynamic point verification module is used to extract feature points within the region defined by the motion candidate mask, calculate the optical flow vector of the feature points, and generate RGB image blocks corresponding to the feature points in combination with the current frame image; fuse the RGB image blocks and the optical flow vector to generate fused data blocks, and input the fused data blocks into a preset neural network for dynamic attribute verification to generate dynamic probability values ​​and velocity scalar values; perform threshold filtering based on the dynamic probability values ​​to obtain a verified dynamic point set. The motion feature clustering module is used to calculate the corresponding optical flow direction angle for each point in the verified dynamic point set; generate a fused velocity value based on the optical flow vector and the velocity scalar value; construct a motion feature vector by combining the position coordinates of each point in the verified dynamic point set and the optical flow direction angle; and perform motion consistency clustering processing based on the motion feature vector to generate the center position coordinates, average velocity value, and dominant direction angle of the obstacle instance. The global motion estimation and filtering module is used to construct robot coordinate system velocity components based on the dominant direction angle and the average velocity value, construct a rotation matrix based on the robot heading angle, perform coordinate transformation on the robot coordinate system velocity components to generate global velocity components, and perform Kalman filtering processing in combination with the center position coordinates to generate smooth position estimates and smooth velocity estimates. The threat calculation module is used to perform Euclidean distance calculation based on the smoothed position estimate and the position coordinates in the current robot pose to generate the relative distance to the obstacle; and to perform threat calculation based on the smoothed velocity estimate and the relative distance to the obstacle to generate a dynamic threat value.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images