Visual recognition obstacle detection method, device, equipment and storage medium

Abstract

The present application relates to the field of visual recognition technology, and discloses a visual recognition obstacle detection method, device, equipment and storage medium, the method comprising: performing time alignment and spatial registration on image data collected by a visual camera in a walking robot and point cloud data collected by a laser radar to obtain a multimodal data stream; inputting the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain temporal multimodal fusion features; calculating an inter-modal correlation matrix, and constructing a three-dimensional voxel grid to obtain a three-dimensional environment representation; performing two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation; performing environmental complexity assessment based on the comprehensive obstacle representation, and generating an adaptive real-time obstacle avoidance control strategy; converting the adaptive real-time obstacle avoidance control strategy into an optimal control problem, and performing rolling time domain optimization to obtain an optimal obstacle avoidance control sequence, thereby improving the accuracy of obstacle detection.

Description

Technical Field

[0001] The present application relates to the field of visual recognition technology, and in particular to a visual recognition obstacle detection method, device, equipment and storage medium. Background Art

[0002] Walking robots need to have accurate environmental perception and obstacle detection capabilities to ensure safe use. Traditional single-sensor obstacle detection methods face many challenges in complex indoor and outdoor environments and are difficult to meet the requirements of high precision and real-time performance.

[0003] In a dynamic environment, how to accurately distinguish between static and dynamic obstacles and predict their motion trajectories is crucial to achieving safe and reliable obstacle avoidance control. Considering the special usage scenarios of assisted mobility robots, how to ensure detection accuracy while taking into account computing efficiency and energy consumption to achieve adaptive real-time control is an urgent problem to be solved. Summary of the invention

[0004] The present application provides a visual recognition obstacle detection method, device, equipment and storage medium, thereby improving the accuracy of obstacle detection.

[0005] The first aspect of the present application provides a visual recognition obstacle detection method, the visual recognition obstacle detection method comprising:

[0006] Perform time alignment and spatial registration on the image data collected by the visual camera and the point cloud data collected by the lidar in the walking robot to obtain a multimodal data stream;

[0007] Inputting the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain temporal multimodal fusion features;

[0008] Based on the temporal multimodal fusion features, an inter-modal correlation matrix is ​​calculated, and a three-dimensional voxel grid is constructed to obtain a three-dimensional environment representation;

[0009] Performing a two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation;

[0010] performing an environment complexity assessment based on the comprehensive obstacle representation and generating an adaptive real-time obstacle avoidance control strategy;

[0011] The adaptive real-time obstacle avoidance control strategy is transformed into an optimal control problem, and a rolling time domain optimization is performed to obtain an optimal obstacle avoidance control sequence of the walking robot.

[0012] A second aspect of the present application provides a visual recognition obstacle detection device, the visual recognition obstacle detection device comprising:

[0013] The acquisition module is used to perform time alignment and spatial registration on the image data collected by the visual camera in the walking robot and the point cloud data collected by the laser radar to obtain a multimodal data stream;

[0014] An extraction module, used for inputting the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain a temporal multimodal fusion feature;

[0015] A calculation module, used to calculate the inter-modality correlation matrix based on the temporal multimodal fusion features, and construct a three-dimensional voxel grid to obtain a three-dimensional environment representation;

[0016] A detection module, configured to perform a two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation;

[0017] A generation module, configured to perform an environment complexity assessment based on the comprehensive obstacle representation and generate an adaptive real-time obstacle avoidance control strategy;

[0018] The optimization module is used to transform the adaptive real-time obstacle avoidance control strategy into an optimal control problem and perform rolling time domain optimization to obtain an optimal obstacle avoidance control sequence for the walking robot.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned visual recognition obstacle detection method.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned visual recognition obstacle detection method.

[0021] Compared with the prior art, the present application has the following beneficial effects: by combining visual camera and lidar data, a more comprehensive and accurate environmental perception is achieved, overcoming the limitations of a single sensor. By adopting a recursive gated deep convolutional neural network, combined with ResNet-50 and PointNet++, efficient feature extraction of image and point cloud data is achieved, and the accuracy of obstacle detection is improved. Through the Kalman filter and motion pattern classifier, static and dynamic obstacles can be effectively distinguished, and their motion trajectories can be predicted, enhancing the prediction ability of the system. Based on the environmental complexity assessment, an adaptive sampling frequency controller and a time-varying high gain event trigger controller are designed to achieve adaptive response to different environments. Through the rolling time domain optimization strategy, the real-time performance of the algorithm is significantly improved while ensuring the control accuracy. The obstacle avoidance problem is transformed into a multi-objective optimization problem, and factors such as energy consumption, safety distance, and trajectory smoothness are comprehensively considered to achieve a more comprehensive optimal control. By adopting efficient numerical methods such as pseudo-spectral method and interior point method, the speed of solving nonlinear programming problems is greatly improved, thereby improving the accuracy of obstacle detection and the accuracy of obstacle avoidance control of the walking robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0024] Figure 1 is a flow chart of a visual recognition obstacle detection method provided by an embodiment of the present invention;

[0025] Figure 2 is a schematic block diagram of the structure of a visual recognition obstacle detection device provided by an embodiment of the present invention;

[0026] Figure 3 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 , an embodiment of the visual recognition obstacle detection method in the embodiment of the present application includes:

[0031] Step 100: performing time alignment and spatial registration on the image data collected by the visual camera and the point cloud data collected by the laser radar in the walking robot to obtain a multimodal data stream;

[0032] It is understandable that the execution subject of the present application may be a visual obstacle detection device, or a terminal or a server, which is not limited here. The present application embodiment is described by taking a server as the execution subject as an example.

[0033] Specifically, the raw image data collected by the visual camera is preprocessed, and the image data is subjected to noise suppression by Gaussian filtering. The image is smoothed by the Gaussian filter to effectively reduce the noise in the image and enhance the image quality. After the Gaussian filter is processed, the noise-suppressed image data is obtained. The noise-suppressed image data is input into the camera intrinsic parameter matrix based on the polynomial model. In order to improve the accuracy of the image, the iterative least squares method is used to estimate and correct the radial distortion and tangential distortion parameters to achieve distortion correction. This method uses the least squares criterion to minimize the error through multiple iterations, calculates the parameters of the lens distortion, and obtains the corrected high-precision image data.

[0034] At the same time, statistical outlier detection is performed on the point cloud data collected by the laser radar in the walking robot. The original point cloud data is filtered by the statistical outlier detection method to remove abnormal points in the point cloud data. The average distance from each point to its K nearest neighbor is calculated, and the average distance of all points is statistically analyzed. When the distance between a point and its neighbors deviates from the overall mean by more than 2 standard deviations, it is judged as an abnormal point and removed to obtain the preliminary filtered point cloud data. The preliminary filtered point cloud data is gridded and divided into several cubic grids of preset sizes. Then, the centroid of the points in each grid is calculated to better represent the local point cloud features. In order to simplify the data and reduce the amount of calculation, voxel filtering is performed to obtain downsampled point cloud data. Voxel filtering greatly reduces the complexity of point cloud data and improves the efficiency of subsequent processing by dividing the space into fixed-size voxel units and calculating the centroid of the points in these voxels. Based on the timestamp information of the corrected image data and the downsampled point cloud data, the cubic spline interpolation algorithm is used to perform time alignment of the data stream to obtain time-aligned multimodal data. Cubic spline interpolation is a smooth interpolation method that accurately interpolates and aligns data without introducing obvious errors, ensuring that image data and point cloud data can be fused and analyzed on the same time basis. Spatial registration of multimodal data. Use the external calibration matrix of the laser radar to convert the camera. Through the external calibration matrix, the point cloud data is projected into the image plane through perspective projection transformation to achieve spatial registration. Perspective projection transformation is a geometric transformation that maps points in three-dimensional space to a two-dimensional image plane, so that the depth information of the laser radar can be combined with the visual information of the camera, and a multimodal data representation that integrates depth and vision is obtained on the image.

[0035] Step 200: Input the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain temporal multimodal fusion features;

[0036] Specifically, the image data in the multimodal data stream is input into the ResNet-50 visual branch for processing. The ResNet-50 visual branch consists of 5 stages, each of which contains multiple residual blocks, and each residual block consists of 3 convolutional layers and a short-circuit connection. The design of the residual block can effectively alleviate the gradient vanishing problem in the deep network and ensure that information and gradients are smoothly propagated in the network. Through layer-by-layer convolution, pooling and activation function (usually ReLU) operations, the network can extract visual features at different scales and obtain multi-scale visual feature maps. At the same time, the point cloud data in the multimodal data stream is input into the PointNet++ lidar branch for processing. The PointNet++ structure contains 4 Set Abstraction layers and 2 Feature Propagation layers. The Set Abstraction layer extracts local features through sampling, grouping and PointNet processing, where the sampling step is used to select representative points, the grouping step divides the point cloud into multiple local areas, and each local area is processed by PointNet to extract features. The Feature Propagation layer restores the resolution of the original point cloud through distance interpolation and unit PointNet processing, while retaining the features extracted from the local area. Through the hierarchical structure, PointNet++ can gradually extract multi-scale point cloud features, so that the spatial information of the lidar can be fully expressed. The spatial attention mechanism is applied to the multi-scale visual feature map and the multi-scale point cloud features. The attention weight of each position in the feature map is calculated to determine which positions are more important for obstacle detection. The attention-enhanced visual features and point cloud features are obtained by weighted summation. The attention-enhanced visual features and point cloud features are spliced, and the spliced ​​features are input into a fully connected layer containing 1024 hidden units, and processed by the ReLU activation function to obtain the first fused feature. The first fused feature is input into the channel attention module for processing. The channel attention module contains global average pooling, global maximum pooling and two fully connected layers. The input features are globally reduced in different ways through global average pooling and global maximum pooling, and then the importance weight of each channel is calculated through two fully connected layers and Sigmoid function. These weights are used to adjust the channels in the features to obtain the second fusion feature that highlights the important channels. Through the channel attention module, different channels are adaptively weighted to effectively highlight the features that contribute most to obstacle detection. The second fusion feature is reshaped into a time series sequence and then input into an LSTM layer containing 512 hidden units. The LSTM layer updates the cell state and hidden state of each time step through the input gate, forget gate, and output gate, extracts the dynamic features in the time series, and obtains the first time series feature.The long short-term memory unit of LSTM can capture the temporal dependency of multimodal features, ensuring that the walking robot can identify obstacles more accurately and timely in a dynamic environment. The self-attention mechanism is applied to the first time series feature, the correlation between different time steps is calculated, and the information association between different time steps is captured, so as to weight the important information in the entire time series and obtain the second time series feature. The second time series feature is input into the gated recurrent unit (GRU), which contains update gates and reset gates. These gates control the flow of information and state updates to obtain time series multimodal fusion features.

[0037] Step 300: Based on the temporal multimodal fusion features, the inter-modal correlation matrix is ​​calculated, and a three-dimensional voxel grid is constructed to obtain a three-dimensional environment representation;

[0038] It should be noted that the temporal multimodal fusion features are reshaped, the visual features and point cloud features are separated, and the similarity between the visual features and the point cloud features is calculated by matrix multiplication to obtain the inter-modal correlation matrix, which measures the degree of connection between the visual features and the point cloud features. The inter-modal correlation matrix is ​​input into the bidirectional feature enhancement module. The bidirectional feature enhancement module contains two parallel attention sub-networks, each of which consists of three fully connected layers and a Softmax layer, which are used to calculate the enhancement weights of the visual features on the point cloud features and the enhancement weights of the point cloud features on the visual features. Through the bidirectional feature enhancement mechanism, the mutual complementation and enhancement between the visual features and the point cloud features are achieved. The weights are applied to the original features by weighted summation to obtain the mutually enhanced features. The mutually enhanced features are input into the multi-scale feature pyramid network. The multi-scale feature pyramid network contains four scale layers, each of which consists of a 3x3 convolution layer, a batch normalization layer, and a ReLU activation function. The feature information of different scales is fused through top-down paths and lateral connections to obtain multi-scale fusion features. The role of the multi-scale feature pyramid network is to effectively fuse features of different resolutions, while capturing the global structure and local details of obstacles, ensuring accurate detection of obstacles of various sizes and shapes. The multi-scale fused features are input into the upsampling module for processing. The upsampling module contains three transposed convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function. The upsampled feature map is obtained by gradually restoring the spatial resolution of the feature map. The upsampling process gradually increases the resolution of the feature map, so that the output feature map has higher spatial details. The initial three-dimensional voxel grid is constructed based on the upsampled feature map. Each pixel in the upsampled feature map is projected into three-dimensional space, and the projected points are assigned to the corresponding voxels to construct a three-dimensional voxel grid. In this way, the two-dimensional visual information is expanded into three-dimensional space, naturally fused with the lidar point cloud features, and a preliminary three-dimensional environment representation is generated. The initial three-dimensional voxel grid is input into the graph convolutional neural network for processing. The graph convolutional neural network contains three graph convolutional layers, each of which uses Chebyshev polynomial approximation for feature aggregation, followed by a batch normalization layer and a LeakyReLU activation function. The graph convolution operation effectively captures the spatial relationship between voxels, and through feature aggregation, the information between adjacent voxels is transmitted to each other, enhancing the spatial representation ability of the three-dimensional voxel grid. After being processed by the graph convolutional neural network, the graph convolution features are obtained. The graph convolution features are residually connected, and the graph convolution features are added to the features of the initial three-dimensional voxel grid, and then the features are fused through a 1x1x1 three-dimensional convolutional layer to obtain the three-dimensional voxel features.The residual connection method can effectively retain the information of the original voxel features, and at the same time, the feature extraction is performed through the fused three-dimensional convolution layer, so that the final three-dimensional voxel features have stronger expressive power. The three-dimensional voxel features are input into the three-dimensional deconvolution network for processing. The three-dimensional deconvolution network contains three three-dimensional deconvolution layers, each of which is followed by a batch normalization layer and a ReLU activation function. By gradually improving the resolution of the voxel grid, a three-dimensional environment representation is obtained.

[0039] Step 400, performing two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation;

[0040] Specifically, the three-dimensional environment representation is input into the region proposal network, and features are extracted by using a 3x3x3 three-dimensional convolution layer, a batch normalization layer, and a ReLU activation function. The convolution layer is used to extract the spatial features of obstacles, the batch normalization layer is used to accelerate training and improve the generalization performance of the model, and the ReLU activation function increases the nonlinear expression ability of the network. Through these steps, the region proposal network can extract rich spatial information and generate candidate regions. The candidate region is then passed through the full convolution layer to generate the specific location of the candidate region, and then the candidate regions with large overlaps are removed by non-maximum suppression operation, and only the most representative candidate regions are retained to obtain preliminary obstacle candidate regions. The preliminary obstacle candidate regions are subjected to RoIAlign (region of interest alignment) operation. The role of RoIAlign is to unify the feature maps of candidate regions of different sizes to a fixed size, maintain the continuity and integrity of spatial information through the bilinear interpolation method, and obtain the normalized candidate region features. Normalization helps to ensure that all candidate regions can be analyzed at the same scale, thereby reducing the deviation caused by different sizes, so that the network can classify and locate obstacles more accurately. The normalized candidate region features are input into a fully connected layer network for processing. The fully connected layer network contains two parallel branches: one is the classification branch and the other is the regression branch. The classification branch outputs the category probability of the obstacle through the Softmax function to determine the specific obstacle type in the candidate region. The regression branch outputs the correction parameters of the bounding box through the linear activation function to refine the boundary of the obstacle and obtain more accurate obstacle detection results. The Kalman filter is applied to the obstacle detection results. The Kalman filter takes the position and velocity of the obstacle as the state vector and estimates the state of the obstacle through prediction and update steps. In the prediction step, it is assumed that the obstacle moves according to the uniform motion model, and its position and velocity are predicted by the prior state estimation; in the update step, the prediction result is corrected by combining the current observed data, and the posterior state estimation is calculated to obtain the filtered obstacle state. Based on the filtered obstacle state, the position sequence, velocity sequence and acceleration sequence of the obstacle are extracted, and the time series information of these sequences is extracted to obtain the time series characteristics of the obstacle. By analyzing the time series features, the probability of the obstacle belonging to static, uniform motion, accelerated motion and irregular motion is calculated to obtain the obstacle motion mode classification result. According to the motion mode classification result, for static obstacles, the original detection results are kept unchanged; for dynamic obstacles, the position is corrected using the prediction results of the Kalman filter to obtain the corrected obstacle position information. The corrected obstacle position information is combined with the original obstacle category information and size information, and the obstacle motion mode information and predicted future trajectory information are added to obtain a comprehensive obstacle representation containing static and dynamic obstacle information.

[0041] Step 500: perform environment complexity assessment based on the comprehensive obstacle representation and generate an adaptive real-time obstacle avoidance control strategy;

[0042] Specifically, based on the comprehensive obstacle representation, the number, density and motion complexity of obstacles in the environment are extracted, and the environmental complexity score is calculated accordingly. The number of obstacles reflects the overall scale of potential dangers in the scene, the obstacle density describes their distribution in space, and the motion complexity is used to evaluate the diversity and uncertainty of the obstacle movement patterns. By combining these factors, a comprehensive environmental complexity score is established to quantify the complexity of the current environment. According to the environmental complexity score, an adaptive sampling frequency controller is designed. The controller maps the environmental complexity score with the preset sampling frequency range and smoothly adjusts the sampling frequency by using the sigmoid function to obtain the optimal sampling frequency. The role of the sigmoid function is to control the change of the sampling frequency to be more continuous and stable, and avoid drastic fluctuations in frequency due to changes in complexity. The higher the environmental complexity, the higher the sampling frequency to ensure that the detection and tracking of obstacles are sufficiently accurate; while in a simple environment, the sampling frequency is appropriately reduced to save system resources and improve operation efficiency. Based on the optimal sampling frequency, the sensor data is resampled. The sensor data is upsampled or downsampled using the linear interpolation method to obtain an adjusted sensor data stream. The adjusted sensor data stream is input into a time-varying high-gain event-triggered controller. The controller contains a state feedback term and a high-gain term, where the high-gain coefficient is proportional to the proximity of the obstacle. This means that when an obstacle approaches the walking robot, the controller will increase the high-gain coefficient to respond to environmental changes more quickly. Through matrix operations, a preliminary control input is obtained, so that the walking robot can adjust its motion state in time to avoid collisions with obstacles. An event trigger function is designed to compare the difference between the current system state and the state at the last trigger. When the difference exceeds the preset threshold, the control update is triggered to obtain a discrete control input sequence. In order to ensure the stability of the system, a parametric Lyapunov function is constructed, and the system state error and control input are used as variables. The Lyapunov function is defined in the form of a quadratic function. The Lyapunov function is used to evaluate the stability of the system. By solving its derivative, the stability condition of the system is determined to ensure the stability of the entire system during dynamic obstacle avoidance. According to the stability conditions of the system, the controller parameters and event trigger thresholds that meet the conditions are calculated by the linear matrix inequality solver to obtain the optimized controller parameters. The linear matrix inequality solver converts the stability conditions of the system into a set of linear matrix inequalities, and finds the optimal control parameters and event trigger thresholds by solving the inequalities. Parameter optimization helps to improve the effectiveness of the control strategy and the response speed of the system, so that the walking robot can flexibly avoid obstacles in complex environments. The optimized controller parameters are substituted into the time-varying high gain event triggered controller, and the adaptive real-time obstacle avoidance control strategy is generated by combining the results of the adaptive sampling frequency and the environmental complexity evaluation.The adaptive real-time obstacle avoidance control strategy can automatically adjust the sampling frequency and control parameters according to environmental changes to adapt to environmental conditions of different complexity.

[0043] Step 600: Convert the adaptive real-time obstacle avoidance control strategy into an optimal control problem, and perform rolling time domain optimization to obtain the optimal obstacle avoidance control sequence of the walking robot.

[0044] Specifically, the state and control variables of the walking robot are modeled. The position, speed and direction of the walking robot are used as state variables, while the acceleration and angular velocity are used as control variables. The kinematic model and dynamic model of the walking robot are constructed, and the system state equation describing its motion is obtained. Based on the system state equation and the adaptive real-time obstacle avoidance control strategy, a multi-objective optimization function is constructed, which includes multiple objectives: minimizing energy consumption, maximizing safety distance, minimizing trajectory smoothness, and minimizing the time to reach the target point. The comprehensive objective function is obtained by balancing each objective by introducing a weight coefficient. Minimizing energy consumption helps to improve the operating efficiency of the walking robot, maximizing the safety distance ensures that the robot can travel safely in an environment with dense obstacles, minimizing trajectory smoothness makes the robot's movement more natural and stable, and minimizing time is used to ensure that the walking robot can quickly reach the target location. By reasonably setting the weights of each objective, a balance between multiple performance indicators is achieved. State constraints and control constraints are added to the comprehensive objective function. State constraints include restrictions on the position, velocity and acceleration of the walking robot to ensure that its movement does not exceed the physical or safety range; while control constraints include restrictions on the maximum steering angle and maximum acceleration to prevent loss of control or unsafe conditions caused by excessive steering or acceleration. By introducing state and control constraints, a constrained optimization problem is obtained, so that the optimization process not only focuses on maximizing or minimizing performance indicators, but also operates within the safety boundary. The constrained optimization problem is discretized in the time dimension, and the continuous-time optimal control problem is transformed into a discrete-time nonlinear programming problem. The pseudo-spectral collocation method is used to expand the state variables and control variables on the Chebyshev polynomial basis function, and the integral term is calculated by the Gauss quadrature formula to obtain an algebraic constrained optimization problem. The pseudo-spectral collocation method can effectively handle complex nonlinear systems by mapping continuous states to a series of discrete collocation points, and improve the accuracy of numerical solutions through Gauss integration. The algebraic constrained optimization problem is transformed into a standard form of nonlinear programming problem, including objective functions, equality constraints and inequality constraints. In order to solve the nonlinear programming problem, the augmented Lagrangian function is constructed to obtain the KKT (Karush-Kuhn-Tucker) condition. The KKT condition is a necessary condition for solving the optimal control problem. By solving these conditions, the optimal control input that meets the objectives and constraints is determined. In order to effectively solve the KKT condition, the interior point method is used as a solution strategy. The interior point method converts inequality constraints into equality constraints and uses barrier functions to ensure that the optimization process is always carried out within the feasible region. The KKT equations are iteratively solved by Newton's method to find the optimal solution. The advantage of the interior point method lies in its numerical stability and good adaptability to high-dimensional nonlinear problems. By gradually weakening the barrier function, it gradually approaches the boundary of the problem and finds the global optimal solution.At the same time, in order to cope with real-time changes in complex dynamic environments, a rolling horizon optimization strategy is adopted. The rolling horizon optimization re-solves the optimization problem every other control cycle, but only executes the optimal control input at the current moment. This strategy allows the constraints to be continuously updated based on the real-time perception of environmental information, so that the control strategy can flexibly respond to changes in the position or motion state of obstacles. In each control cycle, the optimization problem is re-solved based on the latest environmental information collected by the sensor, and the optimal control input at the current moment is obtained, generating the optimal obstacle avoidance control sequence for the walking robot.

[0045] In the embodiment of the present application, by combining visual cameras and laser radar data, a more comprehensive and accurate environmental perception is achieved, overcoming the limitations of a single sensor. A recursive gated deep convolutional neural network is used, combined with ResNet-50 and PointNet++, to achieve efficient feature extraction of image and point cloud data, and improve the accuracy of obstacle detection. Through the Kalman filter and motion pattern classifier, static and dynamic obstacles can be effectively distinguished, and their motion trajectories can be predicted, enhancing the prediction ability of the system. Based on the environmental complexity assessment, an adaptive sampling frequency controller and a time-varying high gain event trigger controller are designed to achieve adaptive response to different environments. Through the rolling time domain optimization strategy, while ensuring the control accuracy, the real-time performance of the algorithm is significantly improved. The obstacle avoidance problem is converted into a multi-objective optimization problem, and factors such as energy consumption, safety distance, and trajectory smoothness are comprehensively considered to achieve a more comprehensive optimal control. The use of efficient numerical methods such as pseudo-spectral method and interior point method greatly improves the speed of solving nonlinear programming problems, thereby improving the accuracy of obstacle detection and the accuracy of obstacle avoidance control of the walking robot.

[0046] In a specific embodiment, the process of executing step 100 may specifically include the following steps:

[0047] Gaussian filtering is performed on the image data collected by the visual camera in the walking robot to obtain noise-suppressed image data, and the noise-suppressed image data is input into the camera intrinsic parameter matrix based on the polynomial model. The radial distortion and tangential distortion parameters are calculated by iterative least squares method, and distortion correction is performed to obtain corrected image data;

[0048] Perform statistical outlier detection on the point cloud data collected by the laser radar in the walking robot, calculate the average distance from each point to its K nearest neighbors, identify points that deviate from the mean by more than 2 standard deviations as abnormal points and remove them, and obtain preliminary filtered point cloud data;

[0049] Divide the initially filtered point cloud data into cubic grids of a preset size, calculate the centroid of each point in the cubic grid, and perform voxel filtering to obtain downsampled point cloud data;

[0050] Based on the timestamp information of the corrected image data and the downsampled point cloud data, the cubic spline interpolation algorithm is used to perform time alignment of the data stream to obtain time-aligned multimodal data;

[0051] Using the external parameter calibration matrix from the lidar to the camera, the point cloud data in the time-aligned multimodal data is projected onto the image plane through perspective projection transformation to obtain a spatially registered multimodal data stream.

[0052] Specifically, for the image data collected by the visual camera, Gaussian filtering is used to suppress the noise in the image. Gaussian filtering is a weight-based smoothing technique that performs a convolution operation on the image through a convolution kernel to smooth the noise, thereby reducing the impact of high-frequency noise. The operation of Gaussian filtering is expressed as:

[0053] ;

[0054] in, is the filtered image data, is the original image at position The pixel value of is the weight of the Gaussian kernel function, defined as:

[0055] ;

[0056] here, is the standard deviation of the Gaussian kernel, which determines the degree of smoothing. Through Gaussian filtering, the noise-suppressed image data is obtained. The noise-suppressed image data is input into the camera intrinsic parameter matrix based on the polynomial model for distortion correction. The camera intrinsic parameter matrix is ​​usually used to convert the two-dimensional image coordinates into a normalized three-dimensional camera coordinate system. For lens distortion, it mainly includes radial distortion and tangential distortion. These distortions are modeled by polynomial models. The common distortion model is expressed as:

[0057] ;

[0058] in, is the normalized coordinate before correction, is the corrected coordinate, are the radial distortion coefficients, and is the tangential distortion coefficient. These parameters are solved by iterative least squares method, so that the image distortion is corrected and the corrected image data is obtained. At the same time, the point cloud data collected by the lidar is processed and statistical outlier detection is performed to remove abnormal points in the data. For each point cloud point, its The average distance between neighboring points, using the mean and standard deviation to identify abnormal points. The average distance between points is , then the points that deviate from the mean by more than 2 standard deviations are judged as abnormal points and removed. Through this step, the point cloud data after preliminary filtering is obtained. The preliminary filtered point cloud data is divided into several cubic grids of preset sizes, and each grid unit contains multiple point cloud points. For each grid, the centroid of the point cloud points is calculated to represent the features within the grid. This operation is voxel filtering. By downsampling the point cloud to fewer points, the amount of data is significantly reduced, thereby improving the computational efficiency, while maintaining the spatial distribution characteristics of the point cloud, and obtaining the downsampled point cloud data. The corrected image data and the downsampled point cloud data are time-aligned. The acquisition time of visual data and lidar data is usually inconsistent, and time alignment is performed to ensure data synchronization. The cubic spline interpolation method is used to effectively achieve time alignment. The timestamp information of the two sets of data is used to interpolate the image and point cloud data through cubic splines, so that they correspond to each other under the same time reference, and obtain time-aligned multimodal data. Using the external parameter calibration matrix from the lidar to the camera, the point cloud data in the time-aligned multimodal data is projected into the image plane through perspective projection transformation. The external calibration matrix describes the spatial relationship between the LiDAR and the camera, that is, the relative position relationship. Through perspective projection transformation, the three-dimensional point cloud data is mapped to the two-dimensional image to achieve spatial registration. The mathematical formula of perspective projection transformation is:

[0059] ;

[0060] in, are the coordinates of the point on the projected image plane, is the camera’s intrinsic parameter matrix, is the external parameter matrix, including the rotation matrix and translation vectors ,and is the point cloud coordinate in three-dimensional space. Through the perspective projection process, the lidar point cloud is matched with the pixel points in the visual image to obtain the registered multimodal data stream.

[0061] In a specific embodiment, the process of executing step 200 may specifically include the following steps:

[0062] The image data in the multimodal data stream is input into the ResNet-50 visual branch. The ResNet-50 visual branch consists of 5 stages, each of which contains multiple residual blocks. Each residual block contains 3 convolutional layers and short-circuit connections. Through layer-by-layer convolution, pooling and activation function operations, a multi-scale visual feature map is obtained.

[0063] Input the point cloud data in the multimodal data stream into the PointNet++ lidar branch. PointNet++ contains 4 Set Abstraction layers and 2 Feature Propagation layers. Each Set Abstraction layer is processed by sampling, grouping and PointNet, and each Feature Propagation layer is processed by distance interpolation and unit PointNet to obtain multi-scale point cloud features.

[0064] Apply the spatial attention mechanism to the multi-scale visual feature map and the multi-scale point cloud features, calculate the attention weight of each position in the feature map, and obtain the attention-enhanced visual features and attention-enhanced point cloud features through weighted summation;

[0065] The attention-enhanced visual features and the attention-enhanced point cloud features are concatenated and input into a fully connected layer containing 1024 hidden units. After being processed by the ReLU activation function, the first fused features are obtained.

[0066] The first fused feature is input into the channel attention module, which includes global average pooling, global maximum pooling and two fully connected layers. The importance weight of each channel is calculated by the Sigmoid function to obtain the second fused feature.

[0067] The second fusion feature is reshaped into a time series sequence and input into an LSTM layer containing 512 hidden units. The LSTM layer updates the cell state and hidden state through the input gate, forget gate, and output gate at each time step to obtain the first time series feature;

[0068] A self-attention mechanism is applied to the first time series feature, the correlation between different time steps is calculated, the second time series feature is obtained by weighted summation, and the second time series feature is input into a gated recurrent unit. The gated recurrent unit includes an update gate and a reset gate. By controlling the information flow and state update, the time series multimodal fusion feature is obtained.

[0069] Specifically, the image data is input into the ResNet-50 visual branch for processing. ResNet-50 is a classic deep convolutional neural network that contains 5 stages, each of which consists of multiple residual blocks. Each residual block contains 3 convolutional layers and a short-circuit connection. The short-circuit connection alleviates the gradient vanishing problem by skipping connections, allowing the network to better train deeper layers. Assume that the input feature is , the output after 3 convolutional layers is , the short-circuit connection adds the input directly to the output, and obtains the residual output

[0070] ;

[0071] in, is the output of the residual block, represents the feature transformation obtained by convolution and activation function, Represents the input of the residual block. ResNet-50 can extract features layer by layer, and combine step-by-step pooling operations and ReLU activation functions to obtain multi-scale visual feature maps that contain image information from different resolutions. At the same time, the point cloud data in the multimodal data stream is input into the PointNet++ lidar branch for processing. PointNet++ is a network for point cloud feature extraction, which contains 4 Set Abstraction layers and 2 Feature Propagation layers. The Set Abstraction layer extracts local features through three steps: sampling, grouping, and PointNet processing. The sampling step uses a sampling algorithm to select representative points from the point cloud. The grouping step divides the points around the sampled points into local regions. The PointNet processing step extracts features from each local region to obtain a feature vector representing the region. Assume that the point set in each local region is , the feature extraction of PointNet is expressed as:

[0072] ;

[0073] in, Indicates the point The nonlinear transformation performed, the max operation means taking the maximum value of the features of all points in the local area, and the obtained is the feature description of the local area. PointNet++ can extract multi-scale point cloud features step by step. The Feature Propagation layer restores the resolution of the original point cloud through distance interpolation and PointNet unit processing, while retaining the local information of the features to obtain multi-scale point cloud features. The spatial attention mechanism is applied to the multi-scale visual feature map and multi-scale point cloud features obtained from ResNet-50 and PointNet++, and the attention weight of each position in the feature map is calculated. The spatial attention mechanism evaluates the importance of different positions in the feature map and weights the features at different positions. Assume that the input feature map is , then the attention weight is expressed as:

[0074] ;

[0075] in, is the position in the feature map The attention weight, is the parameter vector used to calculate the attention weights, Indicates that the feature map is at position feature vector. By weighted summation, we get the attention-enhanced visual features and attention-enhanced point cloud features, which contain enhanced information about obstacles and key parts of the scene. The attention-enhanced visual features and point cloud features are concatenated and input into a fully connected layer containing 1024 hidden units, processed by the ReLU activation function, and the first fused feature is obtained. The first fused feature is input into the channel attention module for processing. The channel attention module performs dimensionality reduction on the fused features through global average pooling and global maximum pooling, and then passes through two fully connected layers, and calculates the importance weight of each channel through the Sigmoid function. Assume that the input feature is , then the channel attention weight w is expressed as:

[0076] ;

[0077] in, represents the Sigmoid function, and is the weight matrix of the fully connected layer, AvgPool and MaxPool Represent the global average pooling and global maximum pooling operations respectively. Through this step, the second fusion feature is obtained. The second fusion feature is reshaped into a time series sequence and input into the LSTM layer containing 512 hidden units. The LSTM (Long Short-Term Memory) network is used to extract the time information in the feature. The cell state and hidden state are updated through the input gate, forget gate and output gate, which can effectively capture the time dependency in the sequence. Through the processing of LSTM, the first time series feature is obtained, which reflects the change information of multimodal data in the time dimension. The self-attention mechanism is applied to the first time series feature to calculate the correlation between different time steps. The self-attention mechanism captures the global time dependency by calculating the relationship between each time step in the sequence. The second time series feature is obtained by weighted summation. The second time series feature is input into the gated recurrent unit (GRU). GRU contains an update gate and a reset gate to control the information flow and state update, and obtains the time series multimodal fusion feature. Compared with LSTM, GRU has fewer parameters and is more lightweight, and can effectively capture the dynamic changes in the time series.

[0078] In a specific embodiment, the process of executing step 300 may specifically include the following steps:

[0079] Reshape the temporal multimodal fusion features, separate them into visual features and point cloud features, calculate the similarity between the visual features and the point cloud features through matrix multiplication, and obtain the inter-modal correlation matrix;

[0080] The inter-modal correlation matrix is ​​input into the bidirectional feature enhancement module. The bidirectional feature enhancement module contains two parallel attention sub-networks. Each attention sub-network consists of three fully connected layers and one Softmax layer. The enhancement weights of visual features on point cloud features and point cloud features on visual features are calculated respectively. Mutual enhancement features are obtained by weighted summation.

[0081] The mutually enhanced features are input into the multi-scale feature pyramid network. The multi-scale feature pyramid network contains four scale layers. Each scale layer consists of a 3x3 convolution layer, a batch normalization layer, and a ReLU activation function. Through top-down paths and lateral connections, feature information of different scales is fused to obtain multi-scale fusion features.

[0082] The multi-scale fusion features are input into the upsampling module. The upsampling module contains three transposed convolutional layers. Each transposed convolutional layer is followed by a batch normalization layer and a ReLU activation function to gradually restore the spatial resolution of the feature map and obtain the upsampled feature map.

[0083] Construct an initial 3D voxel grid based on the upsampled feature map, project each pixel in the upsampled feature map into 3D space, and assign the projection point to the corresponding voxel to obtain a 3D voxel grid;

[0084] The 3D voxel grid is input into the graph convolutional neural network, which contains three graph convolutional layers. Each graph convolutional layer uses Chebyshev polynomial approximation for feature aggregation, followed by a batch normalization layer and a LeakyReLU activation function to capture the spatial relationship between voxels and obtain graph convolution features.

[0085] Perform residual connection on the graph convolution features, add the graph convolution features to the features of the initial 3D voxel grid, and then perform feature fusion through a 1x1x1 3D convolution layer to obtain 3D voxel features;

[0086] The 3D voxel features are input into the 3D deconvolution network, which contains three 3D deconvolution layers. Each deconvolution layer is followed by a batch normalization layer and a ReLU activation function to gradually improve the resolution of the voxel grid and obtain a 3D environment representation.

[0087] Specifically, the temporal multimodal fusion features are reshaped and separated into visual features and point cloud features, so as to independently process visual information and point cloud information and analyze the relationship between the two modalities. Assume that the temporal multimodal fusion feature is a matrix , after reshaping, we get the visual feature matrix And the point cloud feature matrix In order to calculate the similarity between the two, matrix multiplication is used to calculate the similarity between visual features and point cloud features to obtain the inter-modal correlation matrix

[0088] ;

[0089] in, represents the correlation matrix between modes, and They are visual feature and point cloud feature matrices respectively, is the transpose of the point cloud feature. Through matrix multiplication, Each element in represents the similarity between the visual feature and the point cloud feature. The inter-modal correlation matrix is ​​input into the bidirectional feature enhancement module. The bidirectional feature enhancement module contains two parallel attention sub-networks, each of which consists of three fully connected layers and a Softtmax layer, which are used to calculate the enhancement weights of the visual feature to the point cloud feature and the enhancement weights of the point cloud feature to the visual feature. Assume that the input correlation matrix is , the attention weight is expressed as:

[0090] ;

[0091] in, Represents the attention weight of visual features to point cloud features or point cloud features to visual features, is the weight matrix of the fully connected layer. The attention is nonlinearly transformed and normalized through the ReLU activation function and the Softmax layer. Through weighted summation, mutual enhancement features are obtained. These features comprehensively consider the visual and point cloud information, so that the features of each modality are enhanced under the guidance of the other modality, thereby improving the comprehensive expression ability of information. The mutual enhancement features are input into the multi-scale feature pyramid network. The multi-scale feature pyramid network contains four scale layers, each of which consists of a The multi-scale feature pyramid network is composed of a convolutional layer, a batch normalization layer, and a ReLU activation function. Through the top-down path and lateral connection, the multi-scale feature pyramid network can fuse feature information of different scales to obtain multi-scale fusion features. The input features of the scale layer are , the output after convolution, batch normalization and activation is:

[0092] ;

[0093] in, is the convolution kernel weight, * indicates the convolution operation, BN indicates the batch normalization operation, For the The output features of each scale layer. Through the top-down feature fusion mechanism, the key information at each scale is effectively retained, making the feature representation richer and more comprehensive. The multi-scale fusion features are input into the upsampling module. The upsampling module contains three transposed convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function. The purpose of the upsampling module is to gradually restore the spatial resolution of the feature map and obtain the upsampled feature map. Assume The input features of the transposed convolutional layer are , the feature map obtained after upsampling is:

[0094] ;

[0095] in, is the weight of the transposed convolution kernel, represents the transposed convolution operation, is the upsampled feature map. Through layer-by-layer upsampling, the spatial resolution of the feature map is gradually restored. Construct an initial three-dimensional voxel grid based on the upsampled feature map. Project each pixel in the upsampled feature map into three-dimensional space, and assign the projection points to the corresponding voxel units to form a three-dimensional voxel grid. Voxel units are regarded as small cubes in three-dimensional space, and each small cube represents the feature distribution in a certain spatial area. The three-dimensional voxel grid is input into the graph convolutional neural network for processing. The graph convolutional neural network contains three graph convolutional layers. Each graph convolutional layer uses Chebyshev polynomial approximation to perform feature aggregation, followed by a batch normalization layer and a LeakyReLU activation function. Through the graph convolution operation, the spatial relationship between voxels is effectively captured, and the neighborhood information is aggregated to obtain the graph convolution feature. Assume that the first The input of the layer graph convolution is , then the output after Chebyshev polynomial approximation is expressed as:

[0096] ;

[0097] in, is the output feature of the next layer, is the Laplacian matrix of the graph, is a Chebyshev polynomial, is the learning parameter. In this way, the graph convolution network effectively aggregates the features between voxel grids and captures the structural information in three-dimensional space. In order to improve the feature expression ability, the graph convolution features are residually connected, added to the features of the initial three-dimensional voxel grid, and then passed through a The 3D convolutional layer is used to perform feature fusion to obtain the final 3D voxel features. The residual connection method effectively retains the original feature information, while fusion is performed through 3D convolution to enhance the representation ability of the features. The 3D voxel features are input into the 3D deconvolution network for processing. The 3D deconvolution network contains three 3D deconvolution layers, each of which is followed by a batch normalization layer and a ReLU activation function. The purpose of 3D deconvolution is to gradually increase the resolution of the voxel grid to obtain a 3D environment representation, and finally generate a high-precision 3D representation that includes obstacles in the environment and their spatial distribution.

[0098] In a specific embodiment, the process of executing step 400 may specifically include the following steps:

[0099] The 3D environment representation is input into the region proposal network, features are extracted through a 3x3x3 3D ​​convolutional layer, a batch normalization layer, and a ReLU activation function, candidate regions are generated using a full convolutional layer, and non-maximum suppression is performed on the candidate regions to obtain preliminary obstacle candidate regions;

[0100] Perform RoIAlign operation on the preliminary obstacle candidate regions, unify the feature maps of candidate regions of different sizes to a fixed size, maintain spatial information through bilinear interpolation, and obtain normalized candidate region features;

[0101] The normalized candidate region features are input into the fully connected layer network. The fully connected layer network contains two parallel branches. The classification branch outputs the obstacle category probability through the Softmax function, and the regression branch outputs the bounding box correction parameters through the linear activation function to obtain the obstacle detection result.

[0102] Apply a Kalman filter to the obstacle detection results, taking the position and velocity of the obstacle as the state vector. The prediction step uses the uniform motion model to calculate the prior state estimate, and the update step combines the observed data to calculate the posterior state estimate to obtain the filtered obstacle state.

[0103] Based on the filtered obstacle state, the position sequence, velocity sequence and acceleration sequence of the obstacle are extracted, and the time series information of the position sequence, velocity sequence and acceleration sequence are extracted to obtain the time series characteristics of the obstacle;

[0104] According to the time series characteristics of the obstacle, the probability of the obstacle being static, uniform motion, accelerated motion and irregular motion is calculated to obtain the obstacle motion mode classification result. According to the obstacle motion mode classification result, the original detection result of the static obstacle is maintained, and the position of the dynamic obstacle is corrected using the prediction result of the Kalman filter to obtain the corrected obstacle position information;

[0105] The corrected obstacle position information is combined with the original obstacle category information and size information, and the obstacle's motion mode information and predicted future trajectory information are added to obtain a comprehensive obstacle representation containing static and dynamic obstacle information.

[0106] Specifically, the feature extraction is performed on the 3D environment representation. The 3D environment representation is input into the region proposal network, and a series of The 3D convolution layer, batch normalization layer and ReLU activation function are used to extract input features. The 3D convolution layer is used to capture the features of obstacles in 3D space, the batch normalization layer helps to accelerate the convergence process of the network and alleviate the problem of gradient disappearance, and the ReLU activation function is used to introduce nonlinearity and improve the feature representation ability. Assume that the input 3D environment is represented as , the output after convolution and activation is:

[0107] ;

[0108] in, is the convolution kernel weight, * represents the three-dimensional convolution operation, BN is the batch normalization operation, is the output feature. Through the convolution operation, features that can reflect the potential area of ​​obstacles in the three-dimensional environment are obtained. The candidate region is generated through the full convolution layer, and the region proposal network uses the candidate region to mark the area where obstacles exist in the three-dimensional space. Then the candidate region is non-maximum suppressed, that is, the region with the highest confidence is retained for the candidate region with high overlap, and redundant regions are removed. The process of non-maximum suppression is expressed by the following formula:

[0109] ;

[0110] in, Represents the candidate region and The intersection over union (IoU) is used to measure the degree of overlap between two regions. For candidate regions whose degree of overlap (IoU) exceeds a certain threshold, only the region with the maximum confidence is retained to obtain the preliminary obstacle candidate region. Perform the RolAlign operation on the preliminary obstacle candidate region to unify the feature maps of candidate regions of different sizes to a fixed size. RolAlign aligns the features of the candidate regions through the bilinear interpolation method to maintain the integrity of the spatial information and ensure that all candidate region features have the same size in subsequent processing, so that the network can uniformly process inputs of different sizes during classification and regression. The normalized candidate region features are input into the fully connected layer network for processing. The fully connected layer network contains two parallel branches: one is the classification branch and the other is the regression branch. The classification branch outputs the probability of the obstacle category through the Softmax function, and the regression branch outputs the correction parameters of the bounding box through the linear activation function. Assume that the normalized feature of the input is , the output of the classification branch is expressed as:

[0111] ;

[0112] in, Indicates that the obstacle belongs to The probability of the class, and are the weights and biases of the classification branch. The regression branch is used to output the bounding box correction parameters:

[0113] ;

[0114] in, represents the correction parameters of the bounding box, and are the weights and biases of the regression branch. The obstacle category and its precise position in each candidate area are obtained through the output of the classification and regression branches. A Kalman filter is applied to the obstacle detection results to smooth the position and velocity of the obstacle. The state vector of the Kalman filter contains the position and velocity of the obstacle. In the prediction step, a uniform motion model is used to calculate the prior state estimate. Assume that the current state vector is , the prior state estimate is:

[0115] ;

[0116] in, is the state transfer matrix, is the control matrix, is the control input, is the process noise. The update step combines the observed data to calculate the posterior state estimate and obtain the filtered obstacle state. Based on the filtered obstacle state, the position sequence, velocity sequence and acceleration sequence of the obstacle are extracted, and the time series information of these sequences is extracted to obtain the time series characteristics of the obstacle. The time series characteristics reflect the movement change law of the obstacle in the time dimension, which helps to judge the movement mode of the obstacle. According to the time series characteristics, the probability of the obstacle belonging to static, uniform motion, accelerated motion and irregular motion is calculated to obtain the obstacle's motion mode classification result. Assume that the time series characteristics of the obstacle are , then the obstacle's motion pattern classification is expressed as:

[0117] ;

[0118] in, Indicates that the obstacle belongs to The probability of a motion pattern, and are the weights and biases of motion mode classification. According to the motion mode classification results of the obstacles, for static obstacles, the original detection results are kept unchanged; for dynamic obstacles, the positions are corrected using the prediction results of the Kalman filter to obtain more accurate obstacle position information. The corrected obstacle position information is combined with the original obstacle category information and size information, and the obstacle motion mode information and predicted future trajectory information are added to obtain a comprehensive obstacle representation containing static and dynamic obstacle information.

[0119] In a specific embodiment, the process of executing step 500 may specifically include the following steps:

[0120] Based on the comprehensive obstacle representation, the number, density, and motion complexity of obstacles are extracted, and the environmental complexity score is calculated;

[0121] According to the environmental complexity score, an adaptive sampling frequency controller is designed to map the environmental complexity score with the preset sampling frequency range, and the sampling frequency is smoothly adjusted through the sigmoid function to obtain the optimal sampling frequency;

[0122] Based on the optimal sampling frequency, the sensor data is resampled, and the data is upsampled or downsampled using a linear interpolation method to obtain an adjusted sensor data stream, and the adjusted sensor data stream is input into a time-varying high-gain event trigger controller. The time-varying high-gain event trigger controller includes a state feedback term and a high-gain term, wherein the high-gain coefficient is proportional to the degree of proximity of the obstacle, and a preliminary control input is obtained through matrix operations;

[0123] Design an event trigger function, compare the difference between the current state and the state at the last trigger with the preset threshold, trigger the control update when the difference exceeds the threshold, obtain a discrete control input sequence, and construct a parameter Lyapunov function. Take the system state error and control input as variables, define the Lyapunov function in the form of a quadratic function, and solve its derivative to obtain the system stability condition;

[0124] According to the system stability conditions, the controller parameters and event trigger thresholds that meet the conditions are calculated through the linear matrix inequality solver to obtain the optimized controller parameters. The optimized controller parameters are substituted into the time-varying high gain event triggered controller, and an adaptive real-time obstacle avoidance control strategy is generated by combining the adaptive sampling frequency and environmental complexity evaluation results.

[0125] Specifically, information from the comprehensive obstacle representation is extracted, including the number, density, and motion complexity of obstacles. These parameters reflect the obstacle status in the environment and help to quantitatively evaluate the complexity of the environment. The number of obstacles is counted by the number of individuals in the comprehensive obstacle representation, the obstacle density is calculated by the ratio of the number of obstacles to the volume of space they cover, and the motion complexity is quantified by analyzing the changes in the movement trajectory, speed, and acceleration of the obstacles. Assume that the number of obstacles is , the density is ,in represents the volume of space, while the motion complexity is evaluated by indicators such as the acceleration change rate of obstacles. After combining these factors, the environmental complexity score is obtained by weighted summation.

[0126] ;

[0127] in, is the weight used to weigh different factors. Indicates the motion complexity score, which integrates the speed and acceleration changes of obstacles. The environmental complexity score is used to indicate the dynamic complexity of the current environment. , design an adaptive sampling frequency controller. The goal of this controller is to adaptively adjust the environmental complexity and the sampling frequency of the sensor to ensure that the sampling frequency is high enough to capture environmental changes in complex environments, while reducing the sampling frequency in simple environments to save computing resources. In order to achieve this mapping, the environmental complexity score is linearly mapped to the preset sampling frequency range, and then the sampling frequency is smoothly adjusted using the sigmoid function to obtain the optimal sampling frequency

[0128] ;

[0129] in, and are the minimum and maximum values ​​of the sampling frequency, is the steepness factor of the sigmoid function, is the threshold of the environment complexity score. By using the sigmoid function, the sampling frequency changes slowly when the environment complexity is low, and changes rapidly when the complexity is high, achieving adaptive adjustment to the environment. Based on the optimal sampling frequency , resample the sensor data to obtain the adjusted sensor data stream. If the current sampling frequency is lower than the optimal frequency, upsampling is performed, otherwise downsampling is performed. The data is resampled using the linear interpolation method. Assuming that the original sensor data is , the newly sampled data is ,in is the new number of sampling points, and linear interpolation is expressed as:

[0130] ;

[0131] in, is the time of the new sampling point, and are two sampling points adjacent in time in the original data. In this way, the sampling frequency of the data is adjusted on the basis of maintaining the trend of the original data to obtain a new sensor data stream. The adjusted sensor data stream is input into the time-varying high-gain event-triggered controller, which contains a state feedback term and a high-gain term to adjust the control input according to the obstacle situation in the environment. Among them, the high-gain coefficient is proportional to the degree of proximity of the obstacle, that is, when the obstacle is close, the high-gain coefficient is increased to improve the system response speed. Assume that the state vector is , the high gain factor is ,in is the initial gain, Represents the distance between the obstacle and the walking robot. The control input is expressed as:

[0132] ;

[0133] in, is the state feedback matrix, is the initial control input. Through this step, the controller responds quickly to the proximity of obstacles and improves the effect of obstacle avoidance. In order to optimize the control strategy, an event trigger function is designed to compare the difference between the current system state and the state at the last trigger with the preset threshold. When the state change exceeds the threshold, the control update is triggered to obtain a discrete control input sequence. Assume that the system state is , the state when last triggered was , then the event triggering condition is expressed as:

[0134] ;

[0135] in, is the preset threshold. Through the trigger mechanism, the frequent updates of the controller are reduced, the computational overhead is reduced, and the control input is adjusted quickly when necessary. Construct a parametric Lyapunov function, take the system state error and the control input as variables, and define the Lyapunov function in the form of a quadratic function

[0136] ;

[0137] in, is a positive definite matrix used to measure the degree of state deviation of the system. By solving the derivative of the Lyapunov function, the system stability condition is obtained, that is, the derivative of the Lyapunov function should be less than zero during the operation of the system, thereby ensuring the asymptotic stability of the system. According to the stability condition of the system, the controller parameters and event trigger threshold that meet the conditions are calculated by the linear matrix inequality solver to obtain the optimized controller parameters. The optimized controller parameters are substituted into the time-varying high gain event triggered controller, and the adaptive real-time obstacle avoidance control strategy is generated by combining the adaptive sampling frequency and the environmental complexity evaluation results.

[0138] In a specific embodiment, the process of executing step 600 may specifically include the following steps:

[0139] The position, speed and direction of the walking robot are taken as state variables, and the acceleration and angular velocity are taken as control variables. The kinematic model and dynamic model of the walking robot are constructed to obtain the system state equation.

[0140] Based on the system state equation and the adaptive real-time obstacle avoidance control strategy, a multi-objective optimization function is constructed, including minimizing energy consumption, maximizing safety distance, minimizing trajectory smoothness, and minimizing time to reach the target point. Each objective is balanced through the weight coefficient to obtain a comprehensive objective function.

[0141] Add state constraints and control constraints to the comprehensive objective function, where the state constraints include position, velocity and acceleration constraints, and the control constraints include maximum steering angle and maximum acceleration constraints, to obtain a constrained optimization problem;

[0142] Discretize the constrained optimization problem in the time dimension, transform the continuous-time optimal control problem into a discrete-time nonlinear programming problem, and obtain a discretized nonlinear programming problem. Apply the pseudo-spectral collocation method to the discretized nonlinear programming problem, expand the state variables and control variables on the Chebyshev polynomial basis function, calculate the integral term through the Gauss quadrature formula, and obtain the algebraic constrained optimization problem.

[0143] The algebraic constrained optimization problem is transformed into a standard form of nonlinear programming problem, including the objective function, equality constraints and inequality constraints, and the augmented Lagrangian function is constructed to obtain the KKT condition.

[0144] The interior point method is used to solve the KKT conditions, the inequality constraints are converted into equality constraints through the obstacle function, the KKT equations are iteratively solved using the Newton method, and the rolling time domain optimization strategy is adopted to re-solve the optimization problem in each control cycle. Only the optimal control input at the current moment is executed, and the constraints are updated according to the real-time perceived environmental information. The optimization process is repeated to obtain the optimal obstacle avoidance control sequence of the walking robot.

[0145] Specifically, the position, speed and direction of the walking robot are taken as state variables, and the acceleration and angular velocity are taken as control variables to construct its kinematic and dynamic models and obtain the system state equation. Define the state vector ,in Indicates location, Indicates speed, Indicates the direction angle. At the same time, define the control input vector ,in represents acceleration, represents the angular velocity. The kinematic model of the walking robot is expressed as:

[0146] ;

[0147] in, represents the time derivative of the state vector, is a function that describes how the state of the system evolves over time. The dynamic model is used to describe the acceleration and angular velocity The impact on the system state is calculated to obtain the complete system state equation. Based on the system state equation and the adaptive real-time obstacle avoidance control strategy, a multi-objective optimization function is constructed. The optimization objectives include minimizing energy consumption, maximizing the safety distance, minimizing trajectory smoothness, and minimizing the time to reach the target point. Minimizing energy consumption is achieved by controlling the magnitude of acceleration to reduce power consumption; maximizing the safety distance ensures safety by keeping a long distance from obstacles; minimizing trajectory smoothness aims to reduce drastic changes in control input and make the movement smoother; and minimizing time ensures that the robot can reach the target position quickly. After combining these objectives, the comprehensive objective function is expressed as:

[0148] ;

[0149] in, represents the comprehensive objective function, represents acceleration, Indicates the safe distance from obstacles. represents the second-order derivative of the position, representing the smoothness of the trajectory, The time required to reach the destination point, is the weight coefficient, which is used to balance the priorities of different optimization objectives. In order to ensure the physical feasibility and safety of the system, state constraints and control constraints are added to the comprehensive objective function. Among them, the state constraints include the limits of position, speed and acceleration, and the control constraints include the limits of maximum steering angle and maximum acceleration. For example, the speed limit is , the acceleration limit is , the maximum steering angle is limited to These constraints ensure that the walking robot will not exceed physical or safety limits in actual operation. Discretize the constrained optimization problem in the time dimension and transform the continuous-time optimal control problem into a discrete-time nonlinear programming problem. Assume that the time interval Divided into Each time step is , then the state equation is expressed in discretized form as:

[0150] ;

[0151] in, and Respectively represent The state and control input of each time step. Through time discretization, the continuous time problem is transformed into a series of discrete time optimization problems. In order to solve the discretized nonlinear programming problem, the pseudo-spectral collocation method is applied. The pseudo-spectral collocation method converts the original optimal control problem into an algebraic constrained optimization problem by expanding the state variables and control variables on the Chebyshev polynomial basis function and using the Gauss quadrature formula to calculate the integral term. Assume that the state variables are expanded as ,in Indicates Chebyshev polynomials, is the corresponding coefficient, then the integral term is approximately calculated by the Gauss quadrature formula to obtain the optimization problem in algebraic form. The algebraic constraint optimization problem is transformed into a standard form of nonlinear programming problem, including the objective function, equality constraints and inequality constraints. In order to solve the nonlinear programming problem, the augmented Lagrangian function is constructed. ,in is the Lagrange multiplier, which is used to process equality and inequality constraints. According to the augmented Lagrangian function, the Karush-Kuhn-Tucker (KKT) condition is obtained, that is, the gradient of the objective function and the gradient of the constraint condition are balanced at the optimal solution:

[0152] ;

[0153] In order to solve the KKT conditions, the interior point method is used. The interior point method converts inequality constraints into equality constraints by introducing a barrier function, and then uses Newton's method to iteratively solve the KKT equations. The interior point method remains inside the feasible domain during the optimization process and gradually approaches the optimal boundary to find the optimal solution. The obstacle parameters are gradually reduced through each iteration to ensure that the final solution satisfies all constraints. During the entire optimization process, a rolling horizon optimization strategy is adopted, that is, the optimization problem is re-solved in each control cycle, but only the optimal control input at the current moment is executed. Rolling horizon optimization can effectively deal with uncertainties in dynamic environments, and use the latest perception data to update constraints and objective functions in each control cycle, thereby generating real-time control inputs. Assume that the current moment is , the rolling horizon optimization is only performed on The optimal control input at time , and continue to solve new optimization problems in the next control cycle. The optimal obstacle avoidance control sequence of the walking robot is obtained.

[0154] The above describes the visual recognition obstacle detection method in the embodiment of the present application. The following describes the visual recognition obstacle detection device 10 in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of the visual recognition obstacle detection device 10 includes:

[0155] The acquisition module 11 is used to perform time alignment and spatial registration on the image data collected by the visual camera in the walking robot and the point cloud data collected by the laser radar to obtain a multimodal data stream;

[0156] An extraction module 12 is used to input the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain a temporal multimodal fusion feature;

[0157] A calculation module 13 is used to calculate the inter-modality correlation matrix based on the temporal multimodal fusion feature, and construct a three-dimensional voxel grid to obtain a three-dimensional environment representation;

[0158] A detection module 14, for performing two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation;

[0159] A generation module 15, for evaluating the complexity of the environment based on the comprehensive obstacle representation and generating an adaptive real-time obstacle avoidance control strategy;

[0160] The optimization module 16 is used to transform the adaptive real-time obstacle avoidance control strategy into an optimal control problem and perform rolling time domain optimization to obtain the optimal obstacle avoidance control sequence of the walking robot.

[0161] Through the synergy of the above components, by combining visual camera and lidar data, a more comprehensive and accurate environmental perception is achieved, overcoming the limitations of a single sensor. The recursive gated deep convolutional neural network is used, combined with ResNet-50 and PointNet++, to achieve efficient feature extraction of image and point cloud data, and improve the accuracy of obstacle detection. Through the Kalman filter and motion pattern classifier, static and dynamic obstacles can be effectively distinguished and their motion trajectories can be predicted, enhancing the prediction ability of the system. Based on the environmental complexity assessment, an adaptive sampling frequency controller and a time-varying high gain event trigger controller are designed to achieve adaptive response to different environments. Through the rolling time domain optimization strategy, the real-time performance of the algorithm is significantly improved while ensuring the control accuracy. The obstacle avoidance problem is transformed into a multi-objective optimization problem, and factors such as energy consumption, safety distance, and trajectory smoothness are comprehensively considered to achieve a more comprehensive optimal control. The efficient numerical methods such as pseudo-spectral method and interior point method are used to greatly improve the speed of solving nonlinear programming problems, thereby improving the accuracy of obstacle detection and the accuracy of obstacle avoidance control of the walking robot.

[0162] See also Figure 3 , Figure 3This is a schematic block diagram of the structure of an electronic device 300 provided in an embodiment of the present application. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected via a device bus 203, wherein the memory 302 may include a non-volatile storage medium and an internal memory.

[0163] The non-volatile storage medium may store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 301, the processor 301 may execute any of the above-mentioned visual recognition obstacle detection methods.

[0164] The processor 301 is used to provide computing and control capabilities to support the operation of the entire electronic device 300 .

[0165] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 301, the processor 301 can execute any of the above-mentioned visual recognition obstacle detection methods.

[0166] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the electronic device 300 involved in the present application scheme. The specific electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0167] It should be understood that the processor 301 may be a central processing unit (CPU), and the processor 301 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0168] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device 300 described above can refer to the corresponding process of the aforementioned visual recognition obstacle detection method, which will not be repeated here.

[0169] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the visual recognition obstacle detection method provided in the embodiment of the present application.

[0170] The computer-readable storage medium may be an internal storage unit of the electronic device 300 in the aforementioned embodiment, such as a hard disk or memory of the electronic device 300. The computer-readable storage medium may also be an external storage device of the electronic device 300, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the electronic device 300.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0173] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A visual recognition obstacle detection method, characterized in that: The method comprises: Perform time alignment and spatial registration on the image data collected by the visual camera and the point cloud data collected by the lidar in the walking robot to obtain a multimodal data stream; Inputting the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain temporal multimodal fusion features; Based on the temporal multimodal fusion features, the inter-modal correlation matrix is ​​calculated, and a three-dimensional voxel grid is constructed, and the three-dimensional voxel grid is input into a graph convolutional neural network, wherein the graph convolutional neural network comprises three graph convolutional layers, each of which uses Chebyshev polynomial approximation for feature aggregation, followed by a batch normalization layer and a LeakyReLU activation function to capture the spatial relationship between voxels and obtain graph convolutional features; residual connection is performed on the graph convolutional features, the graph convolutional features are added to the features of the initial three-dimensional voxel grid, and then feature fusion is performed through a 1x1x1 three-dimensional convolutional layer to obtain three-dimensional voxel features; the three-dimensional voxel features are input into a three-dimensional deconvolutional network, wherein the three-dimensional deconvolutional network comprises three three-dimensional deconvolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function to gradually improve the resolution of the voxel grid and obtain a three-dimensional environment representation; performing a two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation; performing an environment complexity assessment based on the comprehensive obstacle representation and generating an adaptive real-time obstacle avoidance control strategy; The adaptive real-time obstacle avoidance control strategy is transformed into an optimal control problem, and a rolling time domain optimization is performed to obtain an optimal obstacle avoidance control sequence of the walking robot.

2. The visual obstacle detection method according to claim 1, characterized in that: The image data collected by the visual camera in the walking robot and the point cloud data collected by the laser radar are temporally aligned and spatially registered to obtain a multimodal data stream, including: Performing Gaussian filtering on image data collected by a visual camera in a walking robot to obtain image data after noise suppression, and inputting the image data after noise suppression into a camera intrinsic parameter matrix based on a polynomial model, calculating radial distortion and tangential distortion parameters by an iterative least squares method, performing distortion correction, and obtaining corrected image data; Perform statistical outlier detection on the point cloud data collected by the laser radar in the walking robot, calculate the average distance from each point to its K nearest neighbors, determine the points that deviate from the mean by more than 2 standard deviations as abnormal points and remove them, and obtain preliminary filtered point cloud data; Dividing the initially filtered point cloud data into cubic grids of a preset size, calculating the centroid of each point in the cubic grid, and performing voxel filtering to obtain downsampled point cloud data; Based on the timestamp information of the corrected image data and the downsampled point cloud data, a cubic spline interpolation algorithm is used to perform time alignment of the data stream to obtain time-aligned multimodal data; Using the external parameter calibration matrix of the laser radar to the camera, the point cloud data in the time-aligned multimodal data is projected onto the image plane through perspective projection transformation to obtain a spatially registered multimodal data stream.

3. The visual obstacle detection method according to claim 2, characterized in that: The step of inputting the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain a temporal multimodal fusion feature comprises: Inputting the image data in the multimodal data stream into a ResNet-50 visual branch, wherein the ResNet-50 visual branch comprises five stages, each stage comprises a plurality of residual blocks, each residual block comprises three convolutional layers and a short-circuit connection, and obtaining a multi-scale visual feature map through layer-by-layer convolution, pooling, and activation function operations; Inputting the point cloud data in the multimodal data stream into the PointNet++ lidar branch, wherein the PointNet++ includes 4 Set Abstraction layers and 2 Feature Propagation layers, each Set Abstraction layer is processed by sampling, grouping and PointNet, and each Feature Propagation layer is processed by distance interpolation and unit PointNet to obtain multi-scale point cloud features; Applying a spatial attention mechanism to the multi-scale visual feature map and the multi-scale point cloud feature, calculating the attention weight of each position in the feature map, and obtaining an attention-enhanced visual feature and an attention-enhanced point cloud feature by weighted summation; The attention-enhanced visual features and the attention-enhanced point cloud features are concatenated, input into a fully connected layer including 1024 hidden units, and processed by a ReLU activation function to obtain a first fusion feature; Input the first fused feature into a channel attention module, the channel attention module includes global average pooling, global maximum pooling and two fully connected layers, and calculates the importance weight of each channel through a Sigmoid function to obtain a second fused feature; The second fusion feature is reshaped into a time series sequence, and input into an LSTM layer containing 512 hidden units. The LSTM layer updates the cell state and the hidden state through an input gate, a forget gate, and an output gate at each time step to obtain a first time series feature; A self-attention mechanism is applied to the first time series feature, the correlation between different time steps is calculated, the second time series feature is obtained by weighted summation, and the second time series feature is input into a gated recurrent unit, the gated recurrent unit includes an update gate and a reset gate, and the time series multimodal fusion feature is obtained by controlling the information flow and state update.

4. The visual obstacle detection method according to claim 3, characterized in that: The method of calculating the inter-modality correlation matrix based on the temporal multimodal fusion features and constructing a three-dimensional voxel grid to obtain a three-dimensional environment representation includes: Reshape the temporal multimodal fusion features, separate the temporal multimodal fusion features into visual features and point cloud features, calculate the similarity between the visual features and the point cloud features by matrix multiplication, and obtain an inter-modal correlation matrix; Inputting the inter-modal correlation matrix into a bidirectional feature enhancement module, the bidirectional feature enhancement module comprises two parallel attention sub-networks, each of which consists of three fully connected layers and one Softmax layer, respectively calculating the enhancement weights of visual features on point cloud features and the enhancement weights of point cloud features on visual features, and obtaining mutual enhancement features by weighted summation; Inputting the mutual enhancement features into a multi-scale feature pyramid network, the multi-scale feature pyramid network comprises four scale layers, each scale layer consists of a 3x3 convolution layer, a batch normalization layer and a ReLU activation function, and fusing feature information of different scales through a top-down path and lateral connections to obtain a multi-scale fusion feature; Inputting the multi-scale fusion features into an upsampling module, the upsampling module comprises three transposed convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function, so as to gradually restore the spatial resolution of the feature map and obtain an upsampled feature map; Constructing an initial three-dimensional voxel grid based on the upsampled feature map, projecting each pixel in the upsampled feature map into a three-dimensional space, and assigning the projection points to corresponding voxels to obtain a three-dimensional voxel grid; Inputting the three-dimensional voxel grid into a graph convolutional neural network, the graph convolutional neural network comprises three graph convolutional layers, each graph convolutional layer uses Chebyshev polynomial approximation for feature aggregation, followed by a batch normalization layer and a LeakyReLU activation function to capture the spatial relationship between voxels and obtain graph convolution features; Performing residual connection on the graph convolutional features, adding the graph convolutional features to the features of the initial three-dimensional voxel grid, and then performing feature fusion through a 1x1x1 three-dimensional convolutional layer to obtain three-dimensional voxel features; The three-dimensional voxel features are input into a three-dimensional deconvolution network, which includes three three-dimensional deconvolution layers, each of which is followed by a batch normalization layer and a ReLU activation function, so as to gradually improve the resolution of the voxel grid and obtain a three-dimensional environment representation.

5. The visual recognition obstacle detection method according to claim 4, characterized in that: The performing two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation includes: The three-dimensional environment representation is input into the region proposal network, features are extracted through a 3x3x3 three-dimensional convolution layer, a batch normalization layer, and a ReLU activation function, a candidate region is generated using a full convolution layer, and non-maximum suppression is performed on the candidate region to obtain a preliminary obstacle candidate region; Performing a RoIAlign operation on the preliminary obstacle candidate region, unifying the candidate region feature maps of different sizes into a fixed size, maintaining spatial information through bilinear interpolation, and obtaining normalized candidate region features; Inputting the normalized candidate region features into a fully connected layer network, wherein the fully connected layer network comprises two parallel branches, wherein the classification branch outputs obstacle category probabilities through a Softmax function, and the regression branch outputs bounding box correction parameters through a linear activation function, to obtain obstacle detection results; Applying a Kalman filter to the obstacle detection result, taking the position and velocity of the obstacle as the state vector, calculating a priori state estimation using a uniform motion model in a prediction step, and calculating a posteriori state estimation in combination with observation data in an update step to obtain a filtered obstacle state; Based on the filtered obstacle state, extracting the position sequence, velocity sequence and acceleration sequence of the obstacle, and extracting time series information of the position sequence, velocity sequence and acceleration sequence to obtain the time series characteristics of the obstacle; According to the time series characteristics of the obstacle, the probability of the obstacle belonging to static, uniform motion, accelerated motion and irregular motion is calculated to obtain the motion mode classification result of the obstacle, and according to the motion mode classification result of the obstacle, the original detection result of the static obstacle is maintained, and the position of the dynamic obstacle is corrected using the prediction result of the Kalman filter to obtain the corrected obstacle position information; The corrected obstacle position information is combined with the original obstacle category information and size information, and the obstacle movement mode information and predicted future trajectory information are added to obtain a comprehensive obstacle representation containing static and dynamic obstacle information.

6. The visual obstacle detection method according to claim 5, characterized in that: The step of evaluating the complexity of the environment according to the comprehensive obstacle representation and generating an adaptive real-time obstacle avoidance control strategy comprises: Based on the comprehensive obstacle representation, the number, density, and motion complexity of obstacles are extracted, and an environment complexity score is calculated; According to the environmental complexity score, an adaptive sampling frequency controller is designed to map the environmental complexity score with a preset sampling frequency range, and the sampling frequency is smoothly adjusted through a sigmoid function to obtain an optimal sampling frequency; Based on the optimal sampling frequency, the sensor data is resampled, and the data is upsampled or downsampled using a linear interpolation method to obtain an adjusted sensor data stream, and the adjusted sensor data stream is input into a time-varying high-gain event trigger controller, the time-varying high-gain event trigger controller includes a state feedback term and a high-gain term, wherein the high-gain coefficient is proportional to the degree of proximity of the obstacle, and a preliminary control input is obtained through matrix operation; Design an event trigger function, compare the difference between the current state and the state at the last trigger with the preset threshold, trigger the control update when the difference exceeds the threshold, obtain a discrete control input sequence, and construct a parameter Lyapunov function. Take the system state error and control input as variables, define the Lyapunov function in the form of a quadratic function, and solve its derivative to obtain the system stability condition; According to the system stability condition, the controller parameters and event trigger thresholds that meet the conditions are calculated by a linear matrix inequality solver to obtain optimized controller parameters, and the optimized controller parameters are substituted into the time-varying high gain event trigger controller. Combined with the adaptive sampling frequency and environmental complexity evaluation results, an adaptive real-time obstacle avoidance control strategy is generated.

7. The visual obstacle detection method according to claim 6, characterized in that: The step of converting the adaptive real-time obstacle avoidance control strategy into an optimal control problem and performing rolling time domain optimization to obtain an optimal obstacle avoidance control sequence for the walking robot includes: The position, speed and direction of the walking robot are used as state variables, and the acceleration and angular velocity are used as control variables to construct a kinematic model and a dynamic model of the walking robot, and obtain a system state equation; Based on the system state equation and the adaptive real-time obstacle avoidance control strategy, a multi-objective optimization function is constructed, including a goal of minimizing energy consumption, a goal of maximizing safety distance, a goal of minimizing trajectory smoothness, and a goal of minimizing time to reach the target point, and each goal is balanced by a weight coefficient to obtain a comprehensive objective function; Adding state constraints and control constraints to the comprehensive objective function, wherein the state constraints include restrictions on position, speed and acceleration, and the control constraints include restrictions on maximum steering angle and maximum acceleration, to obtain a constrained optimization problem; Discretizing the constrained optimization problem in the time dimension, converting the continuous-time optimal control problem into a discrete-time nonlinear programming problem, obtaining a discretized nonlinear programming problem, applying a pseudo-spectral collocation method to the discretized nonlinear programming problem, expanding the state variables and the control variables on a Chebyshev polynomial basis function, calculating the integral term by a Gauss quadrature formula, and obtaining an algebraic constrained optimization problem; The algebraic constrained optimization problem is transformed into a standard form of nonlinear programming problem, including an objective function, equality constraints and inequality constraints, and an augmented Lagrangian function is constructed to obtain a KKT condition; The interior point method is used to solve the KKT conditions, the inequality constraints are converted into equality constraints through obstacle functions, the KKT equations are iteratively solved using Newton's method, and a rolling time domain optimization strategy is adopted to re-solve the optimization problem in each control cycle, only executing the optimal control input at the current moment, and updating the constraints according to the real-time perceived environmental information. The optimization process is repeated to obtain the optimal obstacle avoidance control sequence of the walking robot.

8. A visual recognition obstacle detection device, characterized in that: The device is used to perform the visual recognition obstacle detection method according to any one of claims 1 to 7, comprising: The acquisition module is used to perform time alignment and spatial registration on the image data collected by the visual camera in the walking robot and the point cloud data collected by the laser radar to obtain a multimodal data stream; An extraction module, used for inputting the multimodal data stream into a recursive gated deep convolutional neural network for feature extraction to obtain a temporal multimodal fusion feature; A computing module is used to calculate the inter-modal correlation matrix based on the temporal multimodal fusion features, and construct a three-dimensional voxel grid, and input the three-dimensional voxel grid into a graph convolutional neural network, wherein the graph convolutional neural network includes three graph convolutional layers, each of which uses Chebyshev polynomial approximation to perform feature aggregation, followed by a batch normalization layer and a LeakyReLU activation function to capture the spatial relationship between voxels and obtain graph convolutional features; residual connection is performed on the graph convolutional features, the graph convolutional features are added to the features of the initial three-dimensional voxel grid, and then feature fusion is performed through a 1x1x1 three-dimensional convolutional layer to obtain three-dimensional voxel features; the three-dimensional voxel features are input into a three-dimensional deconvolutional network, wherein the three-dimensional deconvolutional network includes three three-dimensional deconvolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function to gradually improve the resolution of the voxel grid and obtain a three-dimensional environment representation; A detection module, configured to perform a two-stage detection on the three-dimensional environment representation to obtain a comprehensive obstacle representation; A generation module, configured to perform an environment complexity assessment based on the comprehensive obstacle representation and generate an adaptive real-time obstacle avoidance control strategy; The optimization module is used to transform the adaptive real-time obstacle avoidance control strategy into an optimal control problem and perform rolling time domain optimization to obtain an optimal obstacle avoidance control sequence for the walking robot.

9. An electronic device, characterized in that: The electronic device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the electronic device executes the visual recognition obstacle detection method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the visual recognition obstacle detection method according to any one of claims 1 to 7 is implemented.

Images