Visual obstacle avoidance method for transfer robot in roll-on-roll-off ship in low-illumination environment

Abstract

The invention belongs to the technical field of transfer robot obstacle avoidance, and particularly relates to a visual obstacle avoidance method for a transfer robot in a roll-on-roll-off ship in a low-illumination environment. The method comprises the steps that multi-source data are collected, the collected multi-source data are preprocessed, and the multi-source data comprise infrared images and laser radar point cloud data; performing target detection on the infrared image, and identifying an obstacle and a type in the infrared image; for the laser radar point cloud data, using an RANSAC plane fitting algorithm to calculate the height of a ground projection, and identifying an obstacle in the laser radar point cloud data; the infrared vision weight and the laser radar weight during fusion are dynamically adjusted according to the ambient illuminance, and obstacles in the infrared image and obstacles in the laser radar point cloud data are fused; and based on the position of the obstacle, the transfer robot avoids the obstacle during path planning.

Description

Technical Field

[0001] The present invention belongs to the technical field of obstacle avoidance for transfer robots, and in particular relates to a visual obstacle avoidance method for a transfer robot in a low-light environment in a roll-on / roll-off ship. Background Art

[0002] In the automated port operation system, the operating scene inside a roll-on / roll-off vessel is extremely unique, with dim lighting, narrow spaces, and complex obstacle distribution. Traditional visual obstacle avoidance systems expose many drawbacks under such low-light conditions, with recognition accuracy significantly reduced and responses significantly delayed, making it difficult to meet actual operational needs. Currently, most existing technologies rely on a single sensor or simple image processing methods. This single technical approach cannot effectively cope with the complex working conditions of low light and multiple obstacles inside a roll-on / roll-off vessel, resulting in poor performance of the visual obstacle avoidance system in actual operation, making it difficult to ensure safe and efficient operations.

[0003] Current technology has shortcomings that urgently need to be addressed. On the one hand, low-light recognition capabilities are severely insufficient, and systemic fault tolerance is lacking. Traditional cameras experience a significant increase in noise and a sharp decrease in image clarity in low-light environments, which in turn reduces obstacle detection accuracy and increases missed detection rates, posing a significant safety hazard to port operations. On the other hand, multi-sensor fusion is inefficient. Existing solutions fail to fully leverage the advantages of infrared vision and lidar. Data fusion algorithms lack a dynamic weight adjustment mechanism, leading to decision delays in complex environments, severely impacting the real-time and smooth operation of operations. Summary of the Invention

[0004] In order to overcome the problems in the prior art, the present invention proposes a visual obstacle avoidance method for a transfer robot in a low-light environment in a roll-on / roll-off ship.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: The present invention provides a visual obstacle avoidance method for a transfer robot in a low-light environment in a roll-on / roll-off ship, comprising the following steps: Step 100: Collect multi-source data and pre-process the collected multi-source data, wherein the multi-source data includes infrared images and lidar point cloud data; Step 200: Target detection is performed on the infrared image to identify obstacles and their types in the infrared image; for the lidar point cloud data, the RANSAC plane fitting algorithm is used to calculate the height of ground protrusions and identify obstacles in the lidar point cloud data; Step 300: Calculate the infrared vision weight and the laser radar weight; for the detected dynamic obstacle, predict the future position of the dynamic obstacle based on the infrared vision weight and the laser radar weight; Step 400: Based on the location of the obstacle, the transfer robot avoids the obstacle when planning the path.

[0006] Furthermore, in step 100, the preprocessing includes: performing denoising on the collected infrared image using a combined algorithm of median filtering and bilateral filtering; and performing voxel grid filtering on the lidar point cloud data to reduce noise.

[0007] Furthermore, in step 200, target detection is performed on the infrared image to identify obstacles and their types in the infrared image, including: establishing an obstacle recognition model based on the YOLOv8 target detection model; during the obstacle recognition model training process, the parameters of the obstacle recognition model are continuously adjusted through the back propagation algorithm, so that the obstacle recognition model can better fit the annotation information in the infrared image dataset, thereby identifying obstacles including pillars, cables, and workers.

[0008] Furthermore, in step 200, the RANSAC plane fitting algorithm is applied to the lidar point cloud data to calculate the height of ground protrusions and identify obstacles in the lidar point cloud data, including: Randomly select at least 3 points from the lidar point cloud data to fit a plane model; Calculate the distance between other points and the plane model, and divide the lidar point cloud into inliers and outliers according to the set distance threshold T; if the distance D of the point to the fitting plane is less than or equal to T, the point is considered an inlier; otherwise, it is considered an outlier; Repeat the process of randomly selecting at least 3 measurement points to fit a plane model and determine the inliers and outliers multiple times, and choose the plane model with the largest number of inliers as the final ground model; By calculating the vertical height difference between the ground point cloud and other point clouds, raised obstacles on the ground are identified.

[0009] Furthermore, in step 300, the infrared vision weight and the lidar weight are calculated, including: ; ; In the above formula, Indicates infrared visual weight; represents the lidar weight; Indicates the ambient light intensity.

[0010] Furthermore, in step 300, for the detected dynamic obstacle, the future position of the dynamic obstacle is predicted based on the infrared vision weight and the lidar weight, including: Calculate the motion vector of dynamic obstacles using the optical flow method based on infrared images; Based on the infrared vision weight and lidar weight, and combined with the lidar velocity measurement value, the extended Kalman filter is used to predict the future position of the obstacle.

[0011] Furthermore, in step 400, based on the location and type of obstacles, the transfer robot avoids obstacles when planning a path, including: Establish a three-degree-of-freedom dynamic model and an environmental model for the transfer robot. The three-degree-of-freedom dynamic model includes a state vector and an input vector. The state vector includes the longitudinal velocity, lateral velocity, and yaw angular velocity of the transfer robot. The input vector includes the rear wheel angle and the additional yaw torque. Considering the obstacle distance constraint, yaw rate deviation constraint, and path deviation constraint, the objective function is established; With the goal of minimizing the comprehensive objective function, the optimization problem is solved under the condition of satisfying the control constraints, that is, the rear wheel angle and additional yaw torque are solved and sent to the actuator of the transfer robot.

[0012] Furthermore, considering the obstacle distance constraint, yaw rate deviation constraint, and path deviation constraint, the objective function J is established, including: ; in, Represents the weight coefficient of the obstacle distance objective function; Represents the weight coefficient of the yaw rate deviation objective function; Represents the weight coefficient of the path deviation objective function; Represents the weight coefficient of the obstacle distance objective function; represents the yaw rate deviation objective function; Represents the path deviation objective function.

[0013] Furthermore, with the goal of minimizing the comprehensive objective function, the optimization problem is solved under the condition of satisfying the control constraints, that is, solving the rear wheel angle and additional yaw moment, including: ; In the above formula, ;in, represents the state vector at time k; Indicates time k The input vector applied to the transfer robot; Indicates time k +1 is the input vector applied to the transfer robot; Indicates at time The control input applied to the transfer robot, where N is the length of the prediction horizon; Represents the state transition function.

[0014] Compared with the prior art, the present invention has the following technical effects: (1) In the low-light, narrow space and complex obstacle operation scenarios inside the port roll-on / roll-off ship, the traditional visual obstacle avoidance system is difficult to cope with complex working conditions due to its reliance on a single sensor or simple image processing method. The method proposed in this invention collects multi-source data such as infrared images and lidar point clouds, and performs effective processing and obstacle identification respectively. Infrared images have advantages in identifying obstacles with heat sources under low illumination, while lidar point clouds can accurately obtain three-dimensional information of objects. The combination of the two overcomes the problems of low recognition accuracy of traditional methods under low illumination and the inability of a single sensor to cope with complex working conditions. It gives full play to the advantages of multiple sensors, provides more comprehensive and accurate obstacle information for the transfer robot, and ensures safe and efficient operation.

[0015] (2) This invention dynamically adjusts the infrared vision weight and lidar weight during fusion based on the ambient light intensity, solving the problems of low multi-sensor fusion efficiency and lack of a dynamic weight adjustment mechanism in existing solutions. In low light conditions, the infrared vision weight is appropriately increased to fully utilize its ability to identify obstacles with heat sources; in high light conditions, the infrared vision weight is reduced and the lidar weight is increased to utilize its advantages in precise three-dimensional measurement. This dynamic adjustment mechanism makes data fusion more in line with the actual environment, reduces decision-making delays, improves the real-time and smoothness of the system in complex environments, and effectively improves the accuracy and timeliness of the transfer robot's obstacle avoidance.

[0016] (3) When planning the obstacle avoidance path based on the location and type of obstacles, the present invention establishes a three-degree-of-freedom dynamic model and an environmental model for the transfer robot, and establishes an objective function by taking into account the obstacle distance constraint, the yaw angular velocity deviation constraint, and the path deviation constraint. By minimizing the comprehensive objective function to solve the optimization problem, the optimal rear wheel steering angle and additional yaw torque control instructions are obtained. This method comprehensively considers the robot's motion characteristics and various constraints in the operating environment, enabling the transfer robot to more reasonably avoid obstacles when planning the path, ensuring the rationality and safety of the path, and further improving the efficiency and reliability of the transfer robot operation within the port roll-on / roll-off ship. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram of the structure of the intelligent transfer robot.

[0019] In the accompanying drawings, 1 is the first axis; 2 is the second axis; 3 is the third axis; 4 is the fourth axis; 5 is the fifth axis; 6 is the hub motor; 7 is the steering motor; 8 is the driving rear wheel; 9 is the clamping mechanism. DETAILED DESCRIPTION

[0020] To further illustrate the technical means and effects employed by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementations, structures, features, and effects of the technical solutions proposed by the present invention. Specific features, structures, or characteristics in one or more embodiments may be combined in any suitable manner. Unless otherwise defined, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention pertains.

[0021] Reference Figure 2 The transfer robot is a five-axis heavy-duty AGV with a five-axis, ten-wheel structure. The first axis 1, the second axis 2, the third axis 3, and the fourth axis 4 are fixed-direction wheels, each of which is independently driven by a hub motor 6; the fifth axis 5 is a power unit equipped with a steering motor 7, which drives the rear wheel 8 to adjust the steering angle; the clamping mechanism 9 is symmetrically arranged on both sides of the center line of the five-axis heavy-duty AGV. When clamping the load, the center of mass is located on the center line, and the longitudinal position changes dynamically. Specifically, the five-axis heavy-duty AGV includes a front plate and a rear plate. The first axis 1 and the second axis 2 of the five-axis heavy-duty AGV are the front plates, equipped with four wheels, each with two motors, but without steering function. The third axis 3 and the fourth axis 4 of the five-axis heavy-duty AGV are the rear plates, equipped with four wheels, each with two motors, but without steering function. Short-range, high-speed wireless communication is used between the front and rear plates of the five-axis heavy-duty AGV to exchange data.

[0022] Reference Figure 1 In this embodiment, a visual obstacle avoidance method for a transfer robot in a low-light environment on a roll-on / roll-off ship is provided, comprising the following steps: Step 100: Collect multi-source data and pre-process the collected multi-source data, wherein the multi-source data includes infrared images and lidar point cloud data; Step 200: Target detection is performed on the infrared image to identify obstacles and their types in the infrared image; for the lidar point cloud data, the RANSAC plane fitting algorithm is used to calculate the height of ground protrusions and identify obstacles in the lidar point cloud data; Step 300: Calculate the infrared vision weight and the lidar weight. For the detected dynamic obstacle, use the extended Kalman filter to predict the future position of the dynamic obstacle based on the infrared vision weight and the lidar weight. Step 400: Based on the location of the obstacle, the transfer robot avoids the obstacle when planning the path.

[0023] The following is a detailed explanation of each of the above steps: Step 100: Collect multi-source data and pre-process the collected multi-source data, wherein the multi-source data includes infrared images and lidar point cloud data.

[0024] As an example, this step 100 specifically includes the following sub-steps: Step 110: Use the infrared camera and the lidar to synchronously obtain the infrared camera video stream and the lidar point cloud data.

[0025] The infrared camera video stream and lidar point cloud data are acquired synchronously at a sampling rate of 200Hz. Specifically, a dedicated synchronization signal generator is used to send trigger signals to both the infrared camera and lidar, ensuring that they begin data acquisition at the same time and continuously and stably at a frequency of 200Hz.

[0026] Hardware timestamp calibration ensures strict temporal alignment between the two data types. Hardware timestamps are precise time stamps automatically added by the hardware during data acquisition. In subsequent data processing and analysis, these timestamps can be used to accurately match infrared images with corresponding LiDAR point cloud data, providing a foundation for multi-source data fusion.

[0027] Step 120: Preprocess the collected infrared camera video stream and lidar point cloud data.

[0028] The preprocessing of the collected infrared camera video stream and lidar point cloud data specifically includes: Step 121: De-noising the collected infrared image using a combination of median filtering and bilateral filtering.

[0029] First, a median filter is performed. This operation iterates over each pixel in the infrared image and replaces its grayscale value with the median grayscale value of its neighboring pixels. Infrared images often contain impulse noise, such as salt and pepper noise, which can severely affect image clarity and subsequent analysis and processing. Median filtering effectively removes this impulse noise. More importantly, while removing noise, it also effectively preserves the image's edge information, maintaining the image's basic contours and features.

[0030] Next, bilateral filtering is performed. Bilateral filtering offers unique advantages in image denoising, preserving image edge details while removing noise. It works by comprehensively considering both the spatial distance and grayscale similarity of pixels. During image smoothing, bilateral filtering assigns less weight to pixels in edge regions, due to the large grayscale differences between them. This cleverly avoids edge blurring and results in sharper, clearer edges.

[0031] The combined use of median and bilateral filtering leverages the strengths of both filtering methods, significantly reducing the noise level of infrared images and significantly improving their quality. This lays a solid foundation for subsequent analysis and processing based on infrared images. Preprocessing of LiDAR point cloud data will also be performed according to specific methods and processes to extract valuable information and optimize data quality.

[0032] Step 122: Perform voxel grid filtering and noise reduction on the lidar point cloud data.

[0033] LiDAR point cloud data usually contains a large number of data points. In order to reduce the data volume and retain key obstacle feature points, a voxel grid filtering downsampling method is used.

[0034] Voxel grid filtering divides the three-dimensional space into small voxel grids, each of which has a certain size. Within each voxel grid, the point cloud data falling within it is processed, usually by calculating the centroid of the points within the grid or randomly selecting a point to represent the point cloud features within the grid. In this way, while ensuring that key obstacle feature points (such as the edges and corners of the obstacle) are retained, the amount of point cloud data is greatly reduced, and the efficiency of subsequent data processing is improved. For example, in autonomous driving scenarios, after voxel grid filtering, obstacle detection and identification can be performed on point cloud data more quickly without affecting the effective perception of the surrounding environment.

[0035] Step 200: Perform target detection on the infrared image to identify obstacles and their types in the infrared image; for the lidar point cloud data, use the RANSAC plane fitting algorithm to calculate the height of ground protrusions and identify obstacles in the lidar point cloud data.

[0036] As an example, step 200 specifically includes: Step 210: Perform target detection on the infrared image to identify obstacles and their types in the infrared image.

[0037] The YOLOv8 model itself is a lightweight object detection model with fast detection speed and relatively high accuracy. During model training, the backpropagation algorithm continuously adjusts the model parameters, enabling the model to better fit the annotation information in the infrared image dataset, accurately identifying 10 types of obstacles, such as pillars, cables, and workers. Common classification loss functions, such as cross-entropy loss, are typically used to measure the difference between the model's predictions and the actual annotations.

[0038] Step 220: For the lidar point cloud data, use the RANSAC plane fitting algorithm to calculate the height of ground protrusions and identify obstacles in the lidar point cloud data.

[0039] Specific examples include: Step 221: Randomly select a small number of points from the lidar point cloud data to fit a plane model, specifically 3 points.

[0040] Step 222: Calculate the distance between other points and the plane model, and divide the lidar point cloud into inliers and outliers according to the set distance threshold T; if the distance D from the point to the fitting plane is less than or equal to T, the point is considered an inlier; otherwise, it is considered an outlier.

[0041] Step 223: Repeat the process of randomly selecting at least 3 measurement points to fit a plane model and determine the inliers and outliers multiple times, assuming the number of iterations is K; and select the plane model with the largest number of inliers as the final ground model.

[0042] Step 224: By calculating the vertical height difference between the ground point cloud and other point clouds, if it is greater than a preset height threshold, it is considered to be a raised obstacle, thereby identifying the raised obstacles on the ground.

[0043] Construct a 3D bounding box of the obstacle and label its type, position, speed and other attributes.

[0044] Step 300: Calculate the infrared vision weight and the lidar weight. For the detected dynamic obstacle, use the extended Kalman filter to predict the future position of the dynamic obstacle based on the infrared vision weight and the lidar weight.

[0045] Obstacle detection in infrared images (based on thermal signatures) is fused with obstacle detection in lidar point cloud data (based on precise 3D spatial information) to combine the strengths of both sensors for more comprehensive and accurate obstacle detection. Infrared images can identify objects with heat, such as pedestrians and animals, but their ability to identify objects without heat sources or with temperatures similar to the surrounding environment, such as rocks and metal railings, is limited. Lidar point clouds can accurately measure the 3D position and shape of objects, but may misdetect or miss objects with similar materials or low reflectivity. Fusion can overcome the limitations of individual sensors and improve the robustness and accuracy of obstacle detection. If the infrared image and lidar point cloud data are in different coordinate systems, coordinate transformation and data alignment are required before fusion. A pre-calibrated transformation matrix can be used to convert obstacle position information from the infrared image to the coordinate system of the lidar point cloud.

[0046] Step 310: Dynamically adjust the infrared vision weight and lidar weight during fusion according to the ambient light intensity: ; ; In the above formula, Indicates infrared visual weight; represents the lidar weight; Indicates the ambient light intensity.

[0047] When ambient light is high, the quality of infrared images may be affected. In this case, the weight of infrared vision can be appropriately reduced and the weight of lidar can be increased. Conversely, when ambient light is low, the advantage of infrared images is more obvious, so the weight of infrared vision can be increased. Specifically, when the ambient light is less than 5 lux, activate the infrared camera's high-gain mode and increase the lidar point cloud confidence weight by 20%.

[0048] Step 320: For the detected dynamic obstacle, calculate its motion vector using the optical flow method, combine it with the velocity measurement value of the lidar, and use the extended Kalman filter to predict the future position of the dynamic obstacle.

[0049] As an example, this step 320 specifically includes: Step 321: Calculate the motion vector of the dynamic obstacle using the optical flow method based on the infrared image.

[0050] Step 321 specifically includes: Step 3211: Preprocess the infrared image.

[0051] For two consecutive frames of infrared images ( t Moment and t +Δ tGrayscale conversion and Gaussian smoothing are performed at each moment to reduce noise interference and retain the grayscale features of the dynamic obstacle area.

[0052] Step 3212: Based on the pre-processed infrared image, establish the basic optical flow equation.

[0053] Assume that the image grayscale function is ,in,( x , y ) represents pixel coordinates, t Indicates time.

[0054] Based on the assumption that image brightness is constant, the grayscale changes of the two frames of images satisfy: ; Performing a first-order Taylor expansion on the left side and ignoring higher-order terms, we get the basic equation of optical flow: ; in, is the spatial grayscale derivative, is the temporal grayscale derivative; Pixels in x 、 y The speed of movement in the direction (motion vector component).

[0055] Step 3213: Solve the basic optical flow equation to obtain the two-dimensional motion vector of the obstacle.

[0056] For the pixel points in the area where the dynamic obstacle is located, the Lucas-Kanade local window method is used to construct an overdetermined equation system and the least squares method is used to solve it. u and v , get the two-dimensional motion vector of the obstacle .

[0057] Step 322: Combine the LiDAR velocity measurements and use an Extended Kalman Filter (EKF) to predict the future position of the obstacle.

[0058] As an example, this step 322 includes: Step 3221: Define the state vector and state transfer function.

[0059] Assume that the state vector of the dynamic obstacle is ,in,( x , y ) is the position coordinate, ( , ) is the velocity component.

[0060] Assuming that the obstacle moves in a uniform straight line, the state transfer function is: ; in, Indicates time k Based on the moment k- 1 The predicted state vector of the information; Indicates time k The state vector of −1 contains the dynamic obstacle at time k- 1. Status information; f () represents the state transition function; 、 Indicates that the obstacle is at time k- 1's position coordinates; Indicates that the obstacle is at time k- The speed of 1 x Directional component; Indicates that the obstacle is at time k- The speed of 1 y The component in the direction; Δt is the sampling time interval, is the process noise (covariance matrix is ​​Q).

[0061] Step 3222: Based on the state transfer function, predict the covariance. The Jacobian matrix F of the state transfer function is: ; Covariance prediction formula: ; in, is the posterior covariance matrix of the previous moment; Represents the current moment of prediction ( k The prior covariance matrix of the time (time); Represents the transposed matrix of the state transition matrix F.

[0062] Step 3223: Based on the infrared vision weight and the lidar weight, the infrared vision measurement value and the lidar measurement value are fused.

[0063] Infrared vision measurement value is (including position and optical flow velocity), the lidar measurement value is (including position and three-dimensional velocity).

[0064] Fused measurements for: ; in, is the weight dynamically adjusted in step 300.

[0065] Step 3224: Based on the predicted current moment ( k The prior covariance matrix of time (time) is used to calculate the Kalman gain.

[0066] Assume that the observation matrix , the observation noise covariance matrix is ​​R, and the Kalman gain is: ; In the above formula, represents the Kalman gain; represents the transpose of the observation matrix.

[0067] Step 3225: State vector update and covariance update.

[0068] Based on the moment k- 1's state vector, Kalman gain, and fused measurement values ​​are used to update the state: ; In the above formula, Indicates time k Based on the moment k The predicted state vector of information.

[0069] Based on the current moment of prediction ( k The prior covariance matrix, Kalman gain, and observation matrix at time t are used to update the covariance: ; In the above formula, Indicates updated k The posterior covariance matrix at time t; I Represents the identity matrix.

[0070] Step 3226: Based on the updated state vector, predict the position in the future (t = 2s): ; In the above formula, 、 They represent the predicted position values ​​at the future time t (here t=2s) based on the updated state vector. x Direction and y Coordinates in direction; x k 、 y k Represents the target or obstacle at time k (current moment) x Direction and y Position coordinates in direction; 、 are the target or obstacle at time k x Direction and y The velocity component in the direction.

[0071] The prediction error is controlled at ≤50mm.

[0072] Step 400: Based on the location of the obstacle, the transfer robot avoids the obstacle when planning the path.

[0073] The MPC (Model Predictive Control) model is used for obstacle avoidance path planning of intelligent transfer robots. Its core idea is to predict and optimize the robot's motion trajectory in the future based on current information to achieve safe and efficient obstacle avoidance.

[0074] Step 410: Establish a three-degree-of-freedom dynamic model and an environmental model of the transfer robot.

[0075] The step 410 specifically includes: Step 411: Construct a three-degree-of-freedom dynamic model of the transfer robot.

[0076] The state vector is defined as the longitudinal velocity, lateral velocity and yaw angular velocity of the transfer robot, that is, Let the state vector ,in, represents the longitudinal speed of the transfer robot; Indicates the lateral speed of the transfer robot; Indicates the yaw angular velocity of the transfer robot.

[0077] The input vector is the rear wheel angle and the additional yaw moment, that is, ,in, is the rear wheel turning angle, is the additional yaw moment.

[0078] Based on the principle of vehicle dynamics, the state equation is: ; in, represents the longitudinal acceleration of the transfer robot; represents the lateral acceleration of the transfer robot; represents the rate of change of the yaw angular velocity of the transfer robot; m is the mass of the transfer robot, is the moment of inertia around the z axis, is the distance from the center of mass to the front axle, is the distance from the center of mass to the rear axle; 、 are the front wheel longitudinal force and the rear wheel longitudinal force respectively; 、 are the lateral forces on the front and rear wheels respectively.

[0079] Step 412: Based on the fused obstacle information, establish an environment model. Based on the fused obstacle information, define the obstacle position as: ; In the above formula, Indicates thei The position vector of each obstacle; 、 Respectively represent i Obstacles in the environment coordinate system x Direction and y The position coordinates of the direction.

[0080] Combined with the future 2-second positions of dynamic obstacles predicted by the extended Kalman filter, an environmental constraint model including fixed obstacles (such as columns and cables) and dynamic obstacles (such as workers) is constructed.

[0081] Step 420: Considering the obstacle distance constraint, the yaw rate deviation constraint, and the path deviation constraint, a comprehensive objective function is established.

[0082] As an example, this step 420 includes: Step 421: Define the sub-objective function.

[0083] The obstacle distance objective function is used to constrain the safe distance between the robot and the obstacle. The formula is: ; in, For the transfer robot and i The distance to the obstacle, is the safety threshold.

[0084] The yaw rate deviation objective function is used to constrain the robot's steering stability. The formula is: ; in, is the desired yaw rate, and the constraints are ( is the maximum yaw angular velocity).

[0085] The path deviation objective function is used to constrain the robot to travel along the desired path. The formula is: ; in, is the actual path, is the desired path, constraining the path deviation .

[0086] Step 422: Construct a comprehensive objective function based on the sub-objective functions J The above sub-objective functions are weighted and combined, and the formula is: ; in, Represents the weight coefficient of the obstacle distance objective function; Represents the weight coefficient of the yaw rate deviation objective function; Represents the weight coefficient of the path deviation objective function. It can be adjusted according to actual needs and scenarios.

[0087] Step 430: Predict the future state and obstacle constraints of the transfer robot.

[0088] Based on the current state and input Use the state equation to predict the state of the next N steps (N is the length of the prediction time domain): ; in, is the state transfer function; the future position constraint of the obstacle is simultaneously incorporated to ensure the distance between the predicted trajectory and the obstacle ; Indicates at time k Predicted, future time k + i The state vector of the transfer robot; Indicates that it is at the time k Predicted, future time k + i- The state vector of the transfer robot at time 1z; Indicates at time k Predicted, future time k + i- 1 is the control input applied to the transfer robot.

[0089] Step 440: Solve the optimization problem with the goal of minimizing the comprehensive objective function while satisfying the control constraints.

[0090] With the goal of minimizing the comprehensive objective function J, under the control constraints (such as the rear wheel angle range , Additional yaw moment range ), solve the optimal control input sequence: ; In the above formula, .in, Indicates time k The input vector applied to the transfer robot; Indicates time k +1 is the input vector applied to the transfer robot; Indicates at time Control input applied to the transfer robot, where N is the length of the prediction horizon.

[0091] Step 450: Execute control instructions and model update.

[0092] As an example, this step 450 includes: Step 451: Execute optimal control.

[0093] Extract the optimal control input sequence obtained by solving the optimization problem The first element of As the control instruction at the current moment, that is, the rear wheel angle and additional yaw moment and sends it to the actuator of the intelligent transfer robot.

[0094] Step 452: Update the model in real time; Every The actual state (position, speed) of the robot and obstacle feedback information are collected, the dynamic model parameters (such as tire cornering stiffness) and the environmental model are modified, and steps 430 to 440 are repeated to generate new control instructions.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A visual obstacle avoidance method for a transfer robot in a low-light environment on a ro-ro ship, characterized in that: The following steps are involved: Step 100: Collect multi-source data and pre-process the collected multi-source data, wherein the multi-source data includes infrared images and lidar point cloud data; Step 200: Target detection is performed on the infrared image to identify obstacles and their types in the infrared image; for the lidar point cloud data, the RANSAC plane fitting algorithm is used to calculate the height of ground protrusions and identify obstacles in the lidar point cloud data; Step 300: Calculate the infrared vision weight and the laser radar weight; for the detected dynamic obstacle, predict the future position of the dynamic obstacle based on the infrared vision weight and the laser radar weight; Step 400: Based on the location of the obstacle, the transfer robot avoids the obstacle when planning the path.

2. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 1, characterized in that: In step 100, the preprocessing includes: performing denoising on the collected infrared image using a combination of median filtering and bilateral filtering; and performing voxel grid filtering on the lidar point cloud data.

3. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 1, characterized in that: In step 200, target detection is performed on the infrared image to identify obstacles and their types in the infrared image, including: establishing an obstacle recognition model based on the YOLOv8 target detection model; during the obstacle recognition model training process, the parameters of the obstacle recognition model are continuously adjusted through the back propagation algorithm so that the obstacle recognition model fits the annotation information in the infrared image dataset, thereby identifying obstacles including pillars, cables, and workers.

4. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 1, characterized in that: In step 200, the RANSAC plane fitting algorithm is applied to the LiDAR point cloud data to calculate the height of ground protrusions and identify obstacles in the LiDAR point cloud data, including: Randomly select at least 3 points from the lidar point cloud data to fit a plane model; Calculate the distance between other points and the plane model, and divide the lidar point cloud into inliers and outliers according to the set distance threshold T; if the distance D of the point to the fitting plane is less than or equal to T, the point is considered an inlier; otherwise, it is considered an outlier; Repeat the process of randomly selecting at least 3 points to fit a plane model and determine the inliers and outliers multiple times, and choose the plane model with the largest number of inliers as the final ground model; By calculating the vertical height difference between the ground point cloud and other point clouds, raised obstacles on the ground are identified.

5. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 1, characterized in that: In step 300, the infrared vision weight and the lidar weight are calculated, including: ； ； In the above formula, Indicates infrared visual weight; represents the lidar weight; Indicates the ambient light intensity.

6. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 5, characterized in that: In step 300, for a detected dynamic obstacle, the future position of the dynamic obstacle is predicted based on the infrared vision weight and the lidar weight, including: Calculate the motion vector of dynamic obstacles using the optical flow method based on infrared images; Based on the infrared vision weight and lidar weight, and combined with the lidar velocity measurement value, the extended Kalman filter is used to predict the future position of the obstacle.

7. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 1, characterized in that: In step 400, based on the location of the obstacle, the transfer robot avoids the obstacle when planning the path, including: Establish a three-degree-of-freedom dynamic model and an environmental model for the transfer robot. The three-degree-of-freedom dynamic model includes a state vector and an input vector. The state vector includes the longitudinal velocity, lateral velocity, and yaw angular velocity of the transfer robot. The input vector includes the rear wheel angle and the additional yaw torque. Considering the obstacle distance constraint, yaw rate deviation constraint, and path deviation constraint, the objective function is established; With the goal of minimizing the comprehensive objective function, the optimization problem is solved under the condition of satisfying the control constraints, that is, the rear wheel angle and additional yaw torque are solved and sent to the actuator of the transfer robot.

8. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 7, characterized in that: Considering the obstacle distance constraint, yaw rate deviation constraint, and path deviation constraint, the objective function J is established, including: ； in, Represents the weight coefficient of the obstacle distance objective function; Represents the weight coefficient of the yaw rate deviation objective function; Represents the weight coefficient of the path deviation objective function; Represents the weight coefficient of the obstacle distance objective function; represents the yaw rate deviation objective function; Represents the path deviation objective function.

9. The visual obstacle avoidance method for a transfer robot in a low-light environment in a ro-ro ship according to claim 8, characterized in that: With the goal of minimizing the comprehensive objective function, the optimization problem is solved under the condition of satisfying the control constraints, that is, solving the rear wheel angle and additional yaw moment, including: ； In the above formula, ;in, Indicates at time k When the state vector; Indicates time k The input vector applied to the transfer robot; Indicates time k +1 is the input vector applied to the transfer robot; Indicates at time The control input applied to the transfer robot, N is the length of the prediction horizon; Represents the state transition function.

Images