AGV obstacle avoidance method and system

Abstract

The invention provides an AGV obstacle avoidance method and system, and the method comprises the steps: carrying out the preprocessing of an original two-dimensional point cloud and an original three-dimensional point cloud collected by a laser radar and a binocular camera in real time, and determining a first target point cloud in the preprocessed two-dimensional point cloud based on the size information of each goods shelf leg, a non-first target point cloud in the preprocessed two-dimensional point cloud is converted into a first point cloud under a vehicle body coordinate system, then the preprocessed three-dimensional point cloud is converted into an initial three-dimensional point cloud under the vehicle body coordinate system, and a second target point cloud in the initial three-dimensional point cloud is deleted based on ground information and size information of shelf legs; the method comprises the steps of obtaining a first point cloud, obtaining a second point cloud, performing obstacle detection on the first point cloud based on a motion state of the AGV, performing obstacle classification on the original RGB image and the second point cloud, and finally determining an obstacle avoidance strategy of the AGV based on an obstacle detection result and an obstacle classification result. According to the invention, the accuracy and flexibility of AGV obstacle avoidance can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AGV obstacle avoidance, and in particular to an AGV obstacle avoidance method and system. Background Art

[0002] AGV (Automated Guided Vehicle) is widely used in fields such as warehousing and logistics. AGV obstacle avoidance is the key to preventing AGV from colliding with obstacles. Currently, the existing AGV obstacle avoidance technologies mainly include the following several solutions:

[0003] (1) Obstacle avoidance technology based on ultrasonic and infrared sensors: The distance between the AGV and the obstacle is measured by emitting and receiving ultrasonic waves through an ultrasonic sensor, and the obstacle is identified by detecting the infrared reflected light through an infrared sensor. Although this solution has the characteristics of low cost and simple implementation, the measurement accuracy of ultrasonic and infrared sensors is relatively low, and it is difficult to accurately identify small-sized or irregularly shaped obstacles, resulting in limited accuracy of this solution. Moreover, this solution is highly dependent on the environment. In a complex environment (such as when there are reflectors or transparent objects), the performance of ultrasonic and infrared sensors will significantly decline. In addition, since the sensors used in this solution can only provide distance information and cannot identify the type of obstacle, this solution lacks semantic information, restricting the flexibility of the obstacle avoidance strategy.

[0004] (2) Obstacle avoidance technology based on lidar: A two-dimensional point cloud map of the environment is created by emitting laser beams through a lidar and measuring the reflection time. Although lidar has high precision and real-time performance, can effectively detect obstacles, and supports complex path planning and obstacle avoidance strategies, lidar is mainly good at detecting static obstacles and has limited processing ability for fast-moving dynamic obstacles (such as pedestrians or vehicles). Moreover, the point cloud data generated by lidar is large in volume and requires powerful computing resources for real-time processing, resulting in complex data processing for this solution.

[0005] (3) Obstacle avoidance technology based on vision sensors: The texture, color, depth, and other information of the environment are captured through vision sensors (such as RGB cameras and depth cameras) to provide rich visual data. Combining traditional computer vision algorithms (such as edge detection and feature matching), the vision sensors are used to detect and track obstacles. Although vision sensors perform particularly well in complex environments and can provide detailed environmental information, vision sensors are highly dependent on the environment. Under low-light or complex lighting conditions, the performance of vision sensors may decline. Moreover, the existing vision sensor technologies mainly rely on traditional computer vision algorithms and cannot provide semantic information about obstacles, restricting the intelligent level of the obstacle avoidance strategy. Summary of the Invention

[0006] In view of this, the objective of the present invention is to provide an AGV obstacle avoidance method and system to alleviate the above problems existing in the existing AGV obstacle avoidance technology.

[0007] In a first aspect, an embodiment of the present invention provides an AGV obstacle avoidance method, including: acquiring the original two-dimensional point cloud collected in real time by a lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by a binocular camera on the AGV, and preprocessing the original two-dimensional point cloud and the original three-dimensional point cloud; wherein, the binocular camera adopts an RGB camera and a depth camera; determining, based on the size information of each shelf leg, a first target point cloud in the preprocessed two-dimensional point cloud that corresponds to a target position and belongs to a first shelf leg area, and converting the non-first target point cloud in the preprocessed two-dimensional point cloud into a first point cloud in the vehicle body coordinate system corresponding to the AGV; wherein, the target position is the position where the AGV is located when the AGV moves to directly below the target shelf; converting the preprocessed three-dimensional point cloud into an initial three-dimensional point cloud in the vehicle body coordinate system, and deleting a second target point cloud belonging to a target area in the initial three-dimensional point cloud based on the ground information and the size information of the shelf leg to obtain a second point cloud; wherein, the target area includes a second shelf leg area and a ground area; performing obstacle detection on the first point cloud based on the motion state of the AGV, and classifying obstacles on the original RGB image and the second point cloud, and then determining an obstacle avoidance strategy for the AGV based on the obstacle detection result and the obstacle classification result.

[0008] In a second aspect, an AGV obstacle avoidance system according to an embodiment of the present invention further includes: an acquisition module, configured to acquire the original two-dimensional point cloud collected in real time by a lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by a binocular camera on the AGV, and preprocess the original two-dimensional point cloud and the original three-dimensional point cloud; wherein, the binocular camera uses an RGB camera and a depth camera; a first processing module, configured to determine, based on the size information of each shelf leg, a first target point cloud in the preprocessed two-dimensional point cloud that corresponds to a target position and belongs to a first shelf leg area, and convert the non-first target point cloud in the preprocessed two-dimensional point cloud into a first point cloud in the vehicle body coordinate system corresponding to the AGV; wherein, the target position is the position where the AGV is located when the AGV moves to directly below the target shelf; a second processing module, configured to convert the preprocessed three-dimensional point cloud into an initial three-dimensional point cloud in the vehicle body coordinate system, and delete a second target point cloud belonging to a target area in the initial three-dimensional point cloud based on ground information and the size information of the shelf leg, to obtain a second point cloud; wherein, the target area includes a second shelf leg area and a ground area; an obstacle avoidance module, configured to perform obstacle detection on the first point cloud based on the motion state of the AGV, classify obstacles in the original RGB image and the second point cloud, and then determine an obstacle avoidance strategy for the AGV based on the obstacle detection result and the obstacle classification result.

[0009] In a third aspect, an electronic device according to an embodiment of the present invention further includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the AGV obstacle avoidance method described in the first aspect above.

[0010] An AGV obstacle avoidance method and system provided by an embodiment of the present invention first obtain the original two-dimensional point cloud collected in real time by a lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by a binocular camera on the AGV, and preprocess the original two-dimensional point cloud and the original three-dimensional point cloud. Then, based on the size information of each shelf leg, the first target point cloud corresponding to the target position and belonging to the first shelf leg area in the preprocessed two-dimensional point cloud is determined, and the non-first target point cloud in the preprocessed two-dimensional point cloud is converted into the first point cloud in the vehicle body coordinate system corresponding to the AGV. Then, the preprocessed three-dimensional point cloud is converted into the initial three-dimensional point cloud in the vehicle body coordinate system, and the second target point cloud belonging to the target area in the initial three-dimensional point cloud is deleted based on the ground information and the size information of the shelf leg to obtain the second point cloud. After that, obstacle detection is performed on the first point cloud based on the motion state of the AGV, and obstacle classification is performed on the original RGB image and the second point cloud. Finally, the obstacle avoidance strategy of the AGV is determined based on the obstacle detection result and the obstacle classification result. By adopting the above technology, lidar data and binocular camera data can be combined for obstacle detection and classification, which not only improves the accuracy of obstacle detection, but also provides the type of obstacle as semantic information, so that the AGV can perform targeted obstacle avoidance according to the type of obstacle, improving the accuracy and flexibility of AGV obstacle avoidance.

[0011] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are realized and obtained by the structures specifically pointed out in the specification, claims, and drawings.

[0012] To make the above objectives, features, and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0014] Figure 1 It is a schematic flowchart of an AGV obstacle avoidance method in an embodiment of the present invention;

[0015] Figure 2 It is an example diagram of dividing rectangular sub-regions in an embodiment of the present invention;

[0016] Figure 3 It is an example diagram of dividing fan-shaped sub-regions in an embodiment of the present invention;

[0017] Figure 4 It is a schematic structural diagram of an AGV obstacle avoidance system in an embodiment of the present invention;

[0018] Figure 5 It is a schematic structural diagram of an electronic device in an embodiment of the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] For ease of understanding of this embodiment, first, a detailed introduction to an AGV obstacle avoidance method disclosed in the embodiments of the present invention is given. Refer to Figure 1 As shown, the method may include the following steps:

[0021] Step S102: Obtain the original two-dimensional point cloud collected in real time by the lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by the binocular camera on the AGV, and preprocess the original two-dimensional point cloud and the original three-dimensional point cloud.

[0022] Among them, the binocular camera can adopt an RGB camera and a depth camera. The original three-dimensional point cloud (generally having X, Y, and Z coordinates in the camera coordinate system) can be collected by the depth camera, and the original RGB image (generally having x and y coordinates in the pixel coordinate system) can be collected by the RGB camera.

[0023] Step S104: Determine, based on the size information of each shelf leg, the first target point cloud in the preprocessed two-dimensional point cloud corresponding to the target position and belonging to the first shelf leg area, and convert the non-first target point cloud in the preprocessed two-dimensional point cloud into the first point cloud in the vehicle body coordinate system corresponding to the AGV.

[0024] Among them, the target position may be the position where the AGV is located when the AGV moves to directly below the target shelf. The size information may be the inner diameter size of the corresponding shelf leg (i.e., the size of the shelf leg along the front-back direction of the AGV and the size of the shelf leg along the left-right direction of the AGV) and the vertical size (i.e., the size of the shelf leg along the vertical direction). The first shelf leg area may be an area expanded according to the inner diameter size and the vertical size of the shelf leg and completely covering the position range where the shelf leg is located.

[0025] Step S106: Convert the preprocessed three-dimensional point cloud into an initial three-dimensional point cloud in the vehicle body coordinate system, and delete the second target point cloud belonging to the target area in the initial three-dimensional point cloud based on the ground information and the size information of the shelf legs to obtain the second point cloud.

[0026] Among them, the target area may include the second shelf leg area and the ground area. The second shelf leg area may be an area obtained by calculating the projection range of the shelf leg in the vehicle body coordinate system according to the inner diameter size and vertical size of the shelf leg and expanding the projection range. The second shelf leg area can completely cover the position range where the shelf leg is located. The ground area is an area expanded from the position range where the points on the ground detected in the initial three-dimensional point cloud are located. The included angle between the normal vector of each point on the ground and the gravity direction is less than a preset angle threshold, and the height value of each point on the ground is less than a preset height threshold.

[0027] Step S108: Perform obstacle detection on the first point cloud based on the motion state of the AGV, classify obstacles for the original RGB image and the second point cloud, and then determine the obstacle avoidance strategy of the AGV based on the obstacle detection result and the obstacle classification result.

[0028] An AGV obstacle avoidance method provided by an embodiment of the present invention first obtains the original two-dimensional point cloud collected in real time by the lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by the binocular camera on the AGV, preprocesses the original two-dimensional point cloud and the original three-dimensional point cloud, then determines the first target point cloud corresponding to the target position and belonging to the first shelf leg area in the preprocessed two-dimensional point cloud based on the size information of each shelf leg, and converts the non-first target point cloud in the preprocessed two-dimensional point cloud into the first point cloud in the vehicle body coordinate system corresponding to the AGV. Then, convert the preprocessed three-dimensional point cloud into an initial three-dimensional point cloud in the vehicle body coordinate system, and delete the second target point cloud belonging to the target area in the initial three-dimensional point cloud based on the ground information and the size information of the shelf legs to obtain the second point cloud. Then, perform obstacle detection on the first point cloud based on the motion state of the AGV, classify obstacles for the original RGB image and the second point cloud, and finally determine the obstacle avoidance strategy of the AGV based on the obstacle detection result and the obstacle classification result. By using the above technology, the lidar data and the binocular camera data can be combined for obstacle detection and classification, which not only improves the accuracy of obstacle detection, but also provides the type of obstacle as semantic information, so that the AGV can perform targeted obstacle avoidance according to the type of obstacle, improving the accuracy and flexibility of AGV obstacle avoidance.

[0029] As a possible implementation manner, the preprocessing of the original two-dimensional point cloud and the original three-dimensional point cloud in the above step S102 may include:

[0030] Step A1: Determine the distance value between each original 2D point in the original 2D point cloud and each of its adjacent points, and compare the distance value corresponding to each original 2D point with a preset distance threshold to delete the isolated points in the original 2D point cloud; wherein, the distance value between each isolated point and each of its adjacent points is greater than the preset distance threshold.

[0031] In the actual application process, the accuracy and reliability of the lidar data (i.e., 2D point cloud data) in the 2D perception system are crucial for subsequent environmental perception and path planning. However, there are often isolated points in the lidar data, which may be generated due to sensor noise, reflection characteristics, or environmental interference, and cause interference to the subsequent processing flow. For lidar data, it is possible to determine whether the current point is an isolated point by comparing the distance value between the current point and its adjacent points; specifically, for each lidar data point, the distance value between the lidar data point and each of its adjacent points can be calculated, and these distance values are compared with a preset distance threshold (such as 30 millimeters or other values). If the distance value between the lidar data point and each of its adjacent points exceeds the preset distance threshold, it can be considered that the lidar data point is an isolated point, and the lidar data point is removed from the lidar data. The judgment threshold for isolated points (i.e., the preset distance threshold) can be selected according to experimental data and the requirements of the actual application scenario to ensure that the judgment threshold for isolated points can filter out isolated points effectively while retaining useful information to the greatest extent.

[0032] Step A2: Perform voxel grid filtering on the original 3D point cloud.

[0033] After obtaining the original point cloud data (i.e., the original 3D point cloud) from the depth camera, these original point cloud data represent the positions of objects in the scene in 3D form, but usually contain a large amount of redundant information and noise. To improve the processing efficiency, it is necessary to perform downsampling processing on these original point cloud data. Specifically, through the voxel grid filtering algorithm, a certain voxel size (such as 0.01m 3 or other sizes) is used to balance the density of the point cloud data and the computational efficiency. The original point cloud data is divided into voxel units of a fixed size, and the representative points within each voxel unit are retained (specifically, randomly selected or designated points within the voxel unit); this process can make the original point cloud data sparser, effectively reduce the amount of point cloud data, and at the same time retain the key geometric features of the corresponding objects (such as shelves, vehicles, etc.).

[0034] As a possible implementation manner, the determination of the first target point cloud corresponding to the target position and belonging to the first shelf leg area in the preprocessed 2D point cloud based on the size information of each shelf leg in the above step S104 may include:

[0035] Step a1: Determine the original coordinates of each shelf leg in its corresponding shelf coordinate system based on the size information of each shelf leg, and convert the original coordinates of each shelf leg into the first coordinates in the laser coordinate system corresponding to the lidar.

[0036] Among them, the original coordinates may include the coordinates of the four corner points and the center point of the corresponding shelf leg.

[0037] In the actual application process, the coordinates of the four corner points and the center point of the shelf leg can be calculated through geometric modeling; specifically, according to the inner diameter size of the shelf leg (i.e., the size of the shelf leg along the front-back direction and the left-right direction of the AGV) and the vertical size, expand the range of the shelf leg (such as expanding 80 mm outward to form a rectangular area that completely surrounds the shelf leg) to ensure that the cut shelf leg area can completely cover the position range where the shelf leg is located during the cutting of the shelf leg. Through geometric calculation, the coordinates of the four shelf legs of each shelf in its corresponding shelf coordinate system are generated (including the coordinates of the four corner points and one center point of each shelf leg in the shelf coordinate system). This way of expanding the range of the shelf leg ensures that even if there are certain manufacturing errors or installation errors in the shelf leg in actual applications, it can be completely covered by the shelf leg area.

[0038] In order to match the position information of the shelf leg with the lidar data, it is necessary to convert the coordinates of the shelf leg from the shelf coordinate system to the lidar coordinate system, and this process is divided into the following two steps:

[0039] 1) Convert the coordinates of the shelf leg from the shelf coordinate system to the vehicle body coordinate system.

[0040] Each shelf can be provided with identification information (such as a QR code, etc.) at the center position of the bottom. When the vehicle body of the AGV moves to directly below the center position of the corresponding shelf bottom, the four shelf legs of the shelf each have corresponding poses (including position and attitude) relative to the vehicle body control center of the AGV (such as the geometric center of the vehicle body); when the AGV is directly below a certain shelf, for each shelf leg of the shelf, based on the pose (x r , y r , θ r ) of the shelf leg relative to the vehicle body control center of the AGV, the coordinates of the shelf leg are converted to the vehicle coordinate system by using the following rigid transformation, and the formula can be expressed as:

[0041]

[0042] Among them, (x car , y car ) represents the position (i.e., coordinates) of the shelf leg in the vehicle body coordinate system corresponding to the AGV, (x rack , y rack) represents the position (i.e., coordinates) of the shelf leg in the corresponding shelf coordinate system, (x r , y r ) and θ r respectively represent the position (i.e., coordinates) and attitude (i.e., angle) of the shelf leg relative to the vehicle body control center of the AGV.

[0043] 2) Convert the coordinates of the shelf leg from the vehicle body coordinate system to the laser coordinate system.

[0044] Since the lidar installed on the vehicle body of the AGV has corresponding poses (including position and attitude) relative to the geometric center of the AGV vehicle body, these parameters can be used as the laser calibration parameters of the AGV; when the AGV is directly below a certain shelf, for each shelf leg of the shelf, after converting the coordinates of the shelf from the shelf coordinate system to the vehicle coordinate system, based on the laser calibration parameters (x l , y l , θ l ) of the AGV, construct an inverse transformation matrix and through the inverse transformation matrix convert the coordinates of the shelf leg from the vehicle body coordinate system to the laser coordinate system according to the following formula:

[0045]

[0046] where, (x laser , y laser ) represents the position (i.e., coordinates) of the shelf leg in the laser coordinate system corresponding to the AGV, (x l , y l ) and θ l respectively represent the position (i.e., coordinates) and attitude (i.e., angle) of the lidar on the AGV relative to the vehicle body control center of the AGV.

[0047] Through this series of coordinate conversions, the position information of the shelf leg can be accurately mapped into the laser coordinate system, aiming to facilitate the subsequent removal of the shelf leg from the original point cloud data obtained by the lidar, so as to avoid the AGV from mis-avoiding obstacles due to the shelf leg of the corresponding shelf being judged as an obstacle when moving directly below the corresponding shelf in the subsequent obstacle avoidance process.

[0048] Step a2, perform angular domain constraint and spatial domain constraint on the first coordinates of each shelf leg to determine the angular range and spatial range of each shelf leg.

[0049] In the actual application process, the coordinates of each shelf leg in the laser coordinate system (including the coordinates of four corner points and one center point) can be formed into a corresponding shelf leg coordinate set, and a polar coordinate constraint model is established for each shelf leg coordinate set in the laser coordinate system. The polar coordinate constraint model includes angular domain constraint and spatial domain constraint.

[0050] Angle domain constraint: Perform polar coordinate transformation on the four corner point coordinates of each shelf leg in the laser coordinate system to convert the coordinates P of each corner point P laser (x, y) in the laser coordinate system into the coordinates P polar (ρ, θ) in the polar coordinate system, and based on the P of the four corner points of each shelf leg polar (ρ, θ), calculate the minimum enclosing angle interval [θ min , θ max of each shelf leg, where θ min and θ max are respectively the minimum and maximum values of θ in the P polar (ρ, θ) of the four corner points of each shelf leg.

[0051] Spatial domain constraint: Synchronously calculate the axis-aligned bounding box of each shelf leg in the Cartesian space to generate a safety boundary {x max +50, x min -50, y max +50, y min -50}, where x max and x min are respectively the maximum and minimum values of the x coordinate of the shelf leg in the laser coordinate system, and y max and y min are respectively the maximum and minimum values of the y coordinate of the shelf leg in the laser coordinate system; specifically, the maximum and minimum values of the x coordinate and y coordinate of each shelf leg in the laser coordinate system can be determined, and an expansion range (±50 mm) is set to expand the coordinates of the shelf leg in the laser coordinate system outward to ensure that the obtained safety boundary {x max +50, x min -50, y max +50, y min -50} can cover the entire area of the shelf leg. This expansion process ensures that the boundaries of the entire area of the shelf leg can be accurately identified in the lidar data.

[0052] Step a3, when the AGV moves directly below the target shelf, obtain the current 2D point cloud corresponding to the current position of the AGV in the preprocessed 2D point cloud, and use the 2D points that are both within the angular range and within the spatial range in the current 2D point cloud as target points to form the first target point cloud.

[0053] After calculating the angular range and spatial range of each shelf leg, enter the stage of lidar data processing. The core task of this stage is to determine whether each lidar data point is within the shelf leg area and perform corresponding processing according to the judgment result. Specifically, the scanned data of the lidar (i.e., the lidar data after preprocessing, which is also the two-dimensional point cloud after preprocessing) can be traversed to determine whether the lidar data point is within the shelf leg area. First, convert the lidar data point in the laser coordinate system into a data point in the polar coordinate system, and compare the angular value of each data point in the polar coordinate system with the angular range of the shelf leg to determine whether the data point is within the angular range of the shelf leg. Second, check the coordinate values of each lidar data point to determine whether each lidar data point is within the spatial range (i.e., the coordinate range) of the shelf leg. Finally, if the angular value of the data point converted from the lidar data point is within the angular range of the shelf leg and the coordinate value of the lidar data point is within the spatial range of the shelf leg, it can be determined that the lidar data point is within the shelf leg area; otherwise (the angular value of the data point converted from the lidar data point is not within the angular range of the shelf leg or the coordinate value of the lidar data point is not within the spatial range of the shelf leg), it can be determined that the lidar data point is not within the shelf leg area. When the AGV moves to directly below a certain shelf, for each lidar data point after preprocessing, if the lidar data point is not within the shelf leg area, then use this lidar data point as the target point, convert the coordinate of this lidar data point from the laser coordinate system to the vehicle body coordinate system corresponding to the AGV, and store the coordinate of this lidar data point in the vehicle body coordinate system. The purpose of converting the lidar data as the target point into the coordinate in the vehicle body coordinate system is to facilitate subsequent obstacle avoidance processing with the vehicle body coordinate system as the core after filtering out the point cloud of the shelf legs, that is, to facilitate judging the positional relationship between the obstacle avoidance point cloud (i.e., the point cloud representing the obstacle) and the vehicle body control center during subsequent obstacle avoidance.

[0054] As a possible implementation manner, the obstacle detection of the first point cloud based on the motion state of the AGV in step S108 may include:

[0055] Step B1, based on the motion state of the AGV, determine the obstacle avoidance area of the AGV and divide the obstacle avoidance area into multiple sub-areas.

[0056] Exemplarily, the motion state may include a straight-line motion state and a stationary rotation motion state. Based on this, the operation mode of the above step B1 may include the following step B11 and step B12:

[0057] Step B11, if the AGV is in the straight-line motion state, determine the obstacle avoidance area as a rectangular obstacle avoidance area, and divide the rectangular obstacle avoidance area into multiple rectangular sub-areas; wherein, each rectangular sub-area corresponds to the corresponding driving range of the AGV.

[0058] Step B12: If the AGV is in a state of rotating in place, determine the obstacle avoidance area as a circular obstacle avoidance area, and divide the circular obstacle avoidance area into multiple sector sub-areas; where each sector sub-area corresponds to a corresponding rotation angle range of the AGV.

[0059] In the actual application process, a dual-mode dynamic switching mechanism for the obstacle avoidance area can be adopted according to the vehicle kinematic characteristics: the obstacle avoidance area mainly has a rectangular area mode and a circular area mode. The rectangular area mode is adapted to the straight-line driving stage of the AGV, and the circular area mode is adapted to the in-place rotation stage of the AGV. Adaptive obstacle perception according to the AGV motion state is achieved through the switching between the rectangular area mode and the circular area mode.

[0060] Rectangular area mode (during the straight-line driving stage of the AGV): When the AGV is in a straight-line motion state, a double-layer rectangular obstacle avoidance area is defined and established based on vehicle kinematic parameters: the double-layer rectangular obstacle avoidance area includes an outer warning area and an inner core area; the outer warning area covers the front and rear space required for the AGV braking distance; the inner core area restricts the lateral safety margin of the AGV to avoid lateral rubbing of the AGV. As shown in Figure 2 As shown, when the AGV is driving straight, a rectangular area 200 can be generated with the vehicle body control center of the AGV as the center as the obstacle avoidance area. The rectangular area 200 is divided into 8 rectangular sub-areas (i.e., the front sub-area 201, the rear sub-area 202, the left sub-area 203, the right sub-area 204, the left front sub-area 205, the right front sub-area 206, the left rear sub-area 207, and the right rear sub-area 208) along the front and rear directions and the left and right directions of the rectangular area 200. The front sub-area 201, the rear sub-area 202, the left front sub-area 205, the right front sub-area 206, the left rear sub-area 207, and the right rear sub-area 208 constitute the outer warning area, and the left sub-area and the right sub-area constitute the inner core area. The two-dimensional point cloud in the vehicle body coordinate system can be projected into these sub-areas through the coordinate axis projection algorithm, so as to quickly segment the two-dimensional point cloud in the vehicle body coordinate system into sub-point clouds corresponding to each sub-area in the subsequent process. This method of determining and dividing the obstacle avoidance area fits the kinematic characteristics of the AGV during the straight-line driving stage, and focuses on enhancing the sensitivity of obstacle detection in the forward direction of the AGV.

[0061] Circular area mode (during the in-place rotation stage of the AGV): As shown in Figure 3As shown, when the AGV is in a state of rotating in place, a circular area 300 with a dynamic sensing radius can be constructed with the centroid of the AGV as the origin as an obstacle avoidance area. The 360° circular area 300 is divided into four 90° fan-shaped sub-areas (i.e., the front sub-area 301, the rear sub-area 302, the left sub-area 303, and the right sub-area 304) using a polar coordinate partitioning strategy, meeting the isotropic obstacle sensing requirements during the rotation of the AGV. This mode eliminates the blind spots of directional preference detection and is conducive to ensuring the omnidirectional safety during the turning of the AGV.

[0062] Step B2: Segment the first point cloud into sub-point clouds corresponding to multiple sub-areas, and perform obstacle detection on each sub-point cloud.

[0063] Exemplarily, the obstacle detection result may include the distance between each detected obstacle and the AGV; based on this, after segmenting the first point cloud into sub-point clouds corresponding to multiple sub-areas, a preset clustering algorithm can be used to cluster each sub-point cloud into corresponding clusters, and the clusters with the number of points corresponding to each sub-point cloud less than the preset point threshold are used as effective clusters representing obstacles; then calculate the minimum distance between each effective cluster and the AGV, and use each obtained minimum distance as the distance between the corresponding obstacle and the AGV.

[0064] In the actual application process, the Fast Euclidean Clustering (FEC) algorithm can be used to cluster the point cloud in each sub-area, and the extracted clusters are used as potential obstacle candidate areas. Set the minimum clustering point threshold N min , and retain the cluster set C = {C1, C2,..., C i} that satisfies |C min |≥N k in each sub-area. Each cluster in this cluster set is an effective cluster representing an obstacle. For each effective cluster C i (i.e., the i-th cluster) corresponding to each sub-area's cluster set C, calculate the minimum Euclidean distance from all points p contained in C i to the vehicle body control center of the AGV and use all the calculated minimum Euclidean distances as the obstacle detection result of this sub-area. Record the positions of the nearest effective obstacles (i.e., effective clusters) in each sub-area to the AGV and the distances between them and the AGV, and compare these distances with the minimum safety distances in each direction (such as the front, rear, left, right, left front, right front, left rear, right rear) of the AGV to provide real-time obstacle avoidance constraints for the subsequent path planning of the AGV through the comparison results.

[0065] As a possible implementation, the obstacle classification of the original RGB image and the second point cloud in the above step S108 may include: extracting features from the original RGB image and the second point cloud using a preset feature extraction network to obtain a first feature corresponding to the original RGB image and a second feature corresponding to the second point cloud; using a pre-trained Convolutional Neural Networks (CNN) to perform classification prediction on the fused features, and taking the classification prediction result as the obstacle classification result; wherein, the classification prediction result includes the classification probabilities of the obstacles corresponding to various categories.

[0066] In the actual application process, ResNet can be selected as the feature extraction network for the RGB image. ResNet is a deep convolutional neural network that solves the problem of gradient disappearance in deep networks by introducing residual connections. The architecture of ResNet includes multiple residual blocks, and each residual block consists of two convolutional layers and an identity mapping. Specifically, the structure of ResNet can include an input layer, convolutional layers, pooling layers, residual layers, global average pooling layers, and fully connected layers. The specific structure is as follows:

[0067] Input layer: Input an RGB image with a size of 224×224×3.

[0068] Convolutional layer: Use a 7×7 convolutional kernel for feature extraction, with a stride of 2 and an output channel number of 64.

[0069] Pooling layer: Use a 3×3 max pooling layer with a stride of 2.

[0070] Residual layer: Contains multiple residual blocks, and each residual block consists of two convolutional layers for feature extraction and residual learning respectively.

[0071] Global average pooling layer: Globally average pool the feature map into a 1×1×512 feature vector.

[0072] Fully connected layer: Input the feature vector and output the classification result.

[0073] For depth data (i.e., three-dimensional point cloud data), PointNet can be selected as the feature extraction network for feature extraction. PointNet is a neural network specifically designed for processing point cloud data, which can directly process unordered point cloud data and maintain invariance to point cloud permutations through symmetric functions. The architecture of PointNet mainly includes the following parts:

[0074] Input layer: Input point cloud data with a size of N×3, where N is the number of points, and 3 represents the x coordinate, y coordinate, and z coordinate of each point.

[0075] Input transformation layer: Align the point cloud by learning the transformation matrix of the input point cloud through a small network T-Net (Transformation Network).

[0076] Feature extraction layer: Use multiple fully connected layers to extract the local features of the point cloud.

[0077] Feature transformation layer: Align the features by learning the transformation matrix of the features through another small network STN (Spatial Transformer Network).

[0078] Symmetric function: Use max pooling to globally aggregate the features to obtain global features.

[0079] Fully connected layer: Input the global features and output the classification results.

[0080] The feature extraction formula of PointNet is as follows:

[0081]

[0082] Among them, is the local feature of the i-th point, MLP is the multi-layer perceptron in the feature extraction layer, N is the total number of points, and max is the symmetric function (used to aggregate local features to obtain the global feature F global .

[0083] Feature fusion is a key step in combining RGB image features (i.e., the above-mentioned feature vectors) and depth data features (i.e., the above-mentioned global features). For RGB image features and depth data features, they can be fused through the following formula:

[0084] F Fused =β·F RGB +(1 - β)·F Depth

[0085] Among them, β is the fusion weight, F Fused is the fused feature, F RGB is the RGB image feature, and F Depth is the depth data feature.

[0086] After obtaining the fused feature, the fused feature can be input into a pre-trained convolutional neural network for classification prediction. The convolutional neural network uses the softmax function as the classifier to output the class probability distribution P=(p1, p2,..., p k ), where p i represents the probability that the obstacle belongs to class i, and the calculation formula of p i is as follows:

[0087]

[0088] where z i is the score of the obstacle belonging to category i (calculated by the convolutional neural network during the classification prediction process).

[0089] The training objective of the model (i.e., the convolutional neural network) is to minimize the classification error to improve the classification accuracy. The cross-entropy loss function can be used as the optimization objective (i.e., using the cross-entropy loss function value to measure the size of the classification error). The cross-entropy loss function can be defined by the following formula:

[0090]

[0091] where M is the number of samples, S is the number of categories, and y ij is the true label (0 or 1) of sample i belonging to category j, and p ij is the probability that the model predicts sample i belongs to category j.

[0092] After obtaining the category probability distribution of the obstacle (including the classification probabilities of the obstacle corresponding to each category) through classification prediction output in the convolutional neural network, the category probability distribution of the obstacle can be used as the obstacle classification result for subsequent use of the obstacle classification result for AGV obstacle avoidance.

[0093] As a possible implementation manner, the above categories may include pedestrians, vehicles, shelves, etc.; based on this, the above step S108 of determining the obstacle avoidance strategy of the AGV based on the obstacle detection result and the obstacle classification result may include the following operation modes:

[0094] Step b1, if the obstacle is a pedestrian, the determined obstacle avoidance strategy of the AGV may include: controlling the AGV to decelerate and bypass the pedestrian when the distance between the pedestrian and the AGV is less than a preset first safety distance; controlling the AGV to select a path away from the pedestrian when planning the path, or controlling the AGV to pause until the distance between the first obstacle and the AGV is not less than the preset first safety distance and then controlling the AGV to continue to travel; adjusting the driving speed of the AGV according to the movement state of the pedestrian so that the distance between the pedestrian and the AGV is greater than the preset first safety distance.

[0095] Step b2. If the obstacle is a vehicle, the determined obstacle avoidance strategy of the AGV may include: If the vehicle is in a stationary state, plan a first obstacle avoidance path based on the size and position of the vehicle and control the AGV to drive along the first obstacle avoidance path to bypass the vehicle; If the vehicle is in a moving state, predict the movement trajectory of the vehicle and adjust the current driving path of the AGV according to the movement trajectory; If the movement trajectory passes through a position point whose distance from the AGV is less than a preset second safety distance, control the AGV to decelerate so that the distance between each position point on the movement trajectory and the AGV is greater than the preset second safety distance; If the movement trajectory coincides with the current driving path of the AGV, control the driving speed of the AGV to be the same as the driving speed of the vehicle.

[0096] Step b3. If the obstacle is a shelf, the determined obstacle avoidance strategy of the AGV may include: Plan a second obstacle avoidance path based on the size and position of the shelf and control the AGV to drive along the second obstacle avoidance path so that the distance between the shelf and the AGV is greater than a preset third safety distance; If the shelf is on the current driving path of the AGV, control the AGV to stop at the current position and re-plan the path, and then control the AGV to drive along the re-planned path so that the distance between the shelf and the AGV is greater than the preset third safety distance; Optimize the current driving path of the AGV to reduce the distance between the optimized path and the target position.

[0097] In the actual application process, given the distance between the known obstacle and the AGV and the type of the obstacle, different obstacle avoidance processing logics can be designed for different types of obstacles (pedestrians, vehicles, shelves) to ensure that the AGV can adopt appropriate obstacle avoidance strategies.

[0098] Obstacle avoidance processing logic for pedestrians: Pedestrians usually have high flexibility and unpredictability, so more cautious obstacle avoidance strategies need to be adopted. Specifically, it can include dynamic avoidance, path planning, and speed adjustment.

[0099] Dynamic avoidance: For pedestrians, set a relatively large safety distance and monitor the movement direction and movement speed of pedestrians in real time. If a pedestrian enters the safety distance range (i.e., the distance between the pedestrian and the AGV is less than the safety distance), the AGV will decelerate and detour (i.e., bypass the pedestrian).

[0100] Path planning: When planning the path, preferentially select a path far from pedestrians. If it is impossible to detour, the robot will pause and wait for the pedestrian to leave the safety area.

[0101] Speed adjustment: Dynamically adjust the driving speed of the AGV according to the movement state of the pedestrian to ensure that the safety distance is always greater than the distance between the pedestrian and the AGV.

[0102] Obstacle avoidance processing logic for vehicles (i.e., vehicles other than the current AGV): Vehicles generally have relatively large sizes and regular motion patterns. Therefore, different obstacle avoidance strategies need to be adopted according to the motion state of the vehicles. Specifically, it can include static vehicle obstacle avoidance strategies and dynamic vehicle obstacle avoidance strategies.

[0103] Static vehicle obstacle avoidance strategy: If the vehicle is stationary, the AGV will plan a detour path based on the size and position of the stationary vehicle. If detouring is not possible, the AGV will pause and wait for the vehicle to leave.

[0104] Dynamic vehicle obstacle avoidance strategy: If the vehicle is moving, the current AGV will predict the motion trajectory of the moving vehicle (which can be achieved through a trajectory planning algorithm), and adjust its current driving path according to the prediction result (i.e., the predicted motion trajectory of the moving vehicle). If it is predicted that the moving vehicle may enter the safe distance range (i.e., there is a position point on the predicted motion trajectory of the moving vehicle whose distance from the current AGV is less than the safe distance), the robot will decelerate and maintain a safe distance. In some cases (such as when the driving path of the current AGV completely or partially coincides with the motion trajectory of the moving vehicle), the current AGV can maintain the same driving speed as the obstacle avoidance vehicle (i.e., the moving vehicle that the current AGV needs to avoid as an obstacle) to avoid frequent acceleration and deceleration of the current AGV.

[0105] Obstacle avoidance processing logic for shelves: Shelves are usually static and have fixed sizes and positions. Therefore, the obstacle avoidance processing logic for shelves is relatively simple. Specifically, it can include the following parts:

[0106] In the path planning stage, the current AGV will ensure that the planned path maintains a sufficient distance from the shelf identified as an obstacle (i.e., the distance between the current AGV and the shelf is less than the set safe distance).

[0107] If a shelf appears suddenly as an obstacle on the current driving path of the current AGV, the current AGV will immediately stop and re-plan the path to ensure that the current AGV will not collide with the shelf subsequently.

[0108] After identifying the target shelf to be carried by the current AGV through semantic information, the current AGV can optimize the path planning and choose to drive on the side closer to the target shelf to improve space utilization.

[0109] For ease of understanding, the implementation method of the above AGV obstacle avoidance method is described exemplarily with a specific application as follows.

[0110] Exemplarily, the above AGV obstacle avoidance method can be implemented according to the following parts:

[0111] (1) 2D laser obstacle avoidance.

[0112] The implementation methods of 2D lidar obstacle avoidance mainly include the following steps 1.1 to step 1.3:

[0113] Step 1.1, lidar data denoising.

[0114] An effective outlier filtering module can be designed to effectively remove the noise points (i.e., outliers) in the lidar data, providing more accurate and reliable input data for subsequent environmental perception and path planning, thus significantly improving the overall performance and robustness of the system.

[0115] Step 1.2, rack leg area cutting.

[0116] This step aims to process the lidar data and filter the lidar data points according to the position information of the rack legs to achieve precise cutting of the rack leg area. Through geometric modeling of the rack legs, coordinate system transformation, and screening of lidar data points, this step can effectively remove the lidar data points within the rack leg area and retain other lidar data points that are effective for AGV obstacle avoidance, providing accurate environmental perception information for subsequent path planning and obstacle avoidance.

[0117] Step 1.2 can include the following steps 1.2.1 to step 1.2.4:

[0118] Step 1.2.1, calculation of rack leg coordinates.

[0119] The calculated rack leg coordinates include the coordinates of the four corner points and a center point of each rack leg in the rack coordinate system.

[0120] Step 1.2.2, coordinate system transformation of the rack leg coordinates.

[0121] When the AGV is directly below a certain rack, the rack leg coordinates of the rack are transformed from the rack coordinate system to the lidar coordinate system to obtain the coordinates of the four corner points and a center point of the rack in the lidar coordinate system.

[0122] Step 1.2.3, polar coordinate modeling of the rack leg area.

[0123] For the coordinates of the four corner points and a center point of each rack leg in the laser coordinate system, a polar coordinate constraint model (including angle domain constraint and spatial domain constraint) is established to calculate the angle range and spatial range of each rack leg.

[0124] Step 1.2.4, lidar data processing.

[0125] Convert the denoised lidar data into data points in the polar coordinate system, and traverse the denoised lidar data and the obtained data points to determine whether the denoised lidar data points are within the shelf leg area according to the angular range and spatial range of each shelf leg, and then convert the lidar data points not within the shelf leg area into target points, transform them into the vehicle body coordinate system and store them.

[0126] Step 1.3, 2D obstacle avoidance perception.

[0127] In this step, a multi-level point cloud segmentation and extreme value detection algorithm is used to realize the perception of obstacles around the vehicle. The core process is divided into three stages: point cloud region division, clustering filtering, and dynamic threshold detection, and supports switching between the rectangular region mode and the circular region mode to meet the obstacle avoidance requirements in different scenarios.

[0128] Since the laser performs 2D obstacle avoidance, the laser point cloud data is first projected onto the two-dimensional plane of the vehicle body coordinate system through coordinate transformation, and then the invalid points (i.e., isolated points) are filtered to retain the valid point cloud.

[0129] Step 1.3 may include the following steps 1.3.1 to step 1.3.2:

[0130] Step 1.3.1, point cloud region division.

[0131] In the rectangular region mode, a rectangular obstacle avoidance region is generated and divided into 8 rectangular sub-regions (as Figure 2 shown); in the circular region mode, a circular obstacle avoidance region is generated and divided into 4 sector-shaped sub-regions (as Figure 3 shown).

[0132] Step 1.3.2, obstacle detection.

[0133] The FEC algorithm is used to cluster the point cloud in each sub-region to extract the clusters corresponding to each sub-region, and the valid clusters with the number of points not less than the minimum clustering point number threshold N min are determined from the clusters corresponding to each sub-region. Then, for each sub-region, the minimum Euclidean distance from each valid cluster corresponding to the sub-region to the AGV vehicle body control center is calculated, and the calculated minimum Euclidean distance corresponding to each sub-region is used as the obstacle detection result of the sub-region.

[0134] (2) 3D camera obstacle avoidance.

[0135] The implementation methods of 3D camera obstacle avoidance mainly include the following steps 2.1 to step 2.5:

[0136] Step 2.1, point cloud data preprocessing.

[0137] Specifically, voxel grid filtering can be performed on the point cloud data obtained from the depth camera to sparsify the point cloud data, which can effectively reduce the amount of point cloud data while retaining key geometric features.

[0138] Step 2.2: Perform coordinate system transformation on the point cloud data.

[0139] To align the point cloud data with the motion state of the AGV, the point cloud data can be transformed from the camera coordinate system to the vehicle body coordinate system. This coordinate system transformation process is achieved through translation and rotation, which can accurately describe the spatial relationship between the camera and the vehicle body. Assume a point P w (X w , Y w , Z w ) is selected in the camera coordinate system. The corresponding coordinates of this point in the vehicle body coordinate system are P c (X c , Y c , Z c ). The transformation relationship between the camera coordinate system and the vehicle body coordinate system can be expressed as:

[0140]

[0141] Among them, R is a 3×3 orthogonal unit rotation matrix, T is a 3×1 translation matrix, and (R, T) is the rigid transformation matrix group for converting from the camera coordinate system to the vehicle body coordinate system.

[0142] The point cloud data can be transformed from the camera coordinate system to the vehicle body coordinate system by using (R, T) through a rigid transformation method, which facilitates subsequent obstacle detection and obstacle avoidance decision-making centered on the vehicle body.

[0143] Step 2.3: Remove the point cloud in the shelf leg area.

[0144] In the actual scenario, the structure of the shelf legs may interfere with the point cloud data and affect the accuracy of obstacle detection. To remove these interferences, the point cloud in the shelf leg area can be removed from the point cloud data through geometric cutting according to the geometric parameters of the shelf legs (such as length, width, and position), thereby retaining the effective obstacle point cloud. This process ensures that the point cloud data for subsequent processing only contains obstacle information related to obstacle avoidance.

[0145] Step 2.4: Remove the ground point cloud.

[0146] The ground point cloud is the main component of the point cloud data but is not directly helpful for the obstacle avoidance function. To remove the ground point cloud, a plane detection method based on normal vectors can be adopted. Specifically, the Random Sample Consensus (RANSAC) algorithm can be used to detect the ground plane from the point cloud, and the ground points can be filtered out and removed according to the angle between the normal vector and the gravity direction. Specifically, the angle between the normal vector of each point in the point cloud and the gravity direction (Z-axis) can be calculated, and the points with an angle between the normal vector and the gravity direction less than the angle threshold (such as 20°) and a height value lower than the preset height threshold are regarded as ground points and removed from the point cloud data. This process can effectively identify and remove the ground points and retain the obstacle point cloud.

[0147] Step 2.5, semantic feature extraction.

[0148] To achieve the semantic classification of obstacles, the RGB camera data and depth camera data can be combined, and a CNN can be used as the deep learning model for obstacle classification. The input of the deep learning model is the fused RGB image and depth data, and the output is the category of the obstacle (such as pedestrians, vehicles, shelves, etc.). ResNet can be used to extract the RGB image features, and PointNet can be used to extract the depth data features. The RGB image provides the texture and color information of the obstacle, while the depth data provides the three-dimensional structure information of the obstacle. By aligning the RGB image with the depth data, we can fuse these two types of data in the same coordinate system, so as to obtain a complete description of the obstacle through the fused RGB image and depth data.

[0149] Before training the deep learning model, dataset annotation is required. Specifically, a large number of datasets containing RGB images and depth data can be collected. The datasets cover a variety of scenarios (including obstacles such as pedestrians, vehicles, and shelves), and each sample in the dataset contains RGB images and depth data; then the samples in the dataset are annotated, and the annotation content includes the category labels of the obstacles (such as pedestrians, vehicles, shelves, etc.) and the positions of the obstacles in the images; the diversity of the data can be increased by performing operations such as rotation, translation, and scaling on the annotated samples, and the samples with annotation content are composed into an annotated dataset.

[0150] When training the deep learning model, the obtained annotated dataset can be divided into a training set (70%), a validation set (15%), and a test set (15%). Then, the training set is used for iterative training of the deep learning model, the validation set is used to verify the effect of the deep learning model to adjust the parameters of the deep learning model, and the test set is used to test the performance of the deep learning model. The cross-entropy loss function can be used as the loss function of the deep learning model.

[0151] Step 2.5 may include the following steps 2.5.1 to 2.5.4:

[0152] Step 2.5.1, RGB feature extraction.

[0153] Step 2.5.2, depth feature extraction.

[0154] Step 2.5.3, multi-modal feature fusion.

[0155] The RGB image feature F Fused and the depth data feature F RGB can be fused into F Depth through the formula F RGB = β·F Depth +(1 - β)·F Fused .

[0156] Step 2.5.4, classifier result output

[0157] The fused feature F Fused can be input into the trained deep learning model for classification prediction, so as to output the class probability distribution of the obstacle as the obstacle classification result through the classifier of the deep learning model.

[0158] (3) 2D + 3D fusion obstacle avoidance.

[0159] "Positioning + recognition" obstacle avoidance can be achieved through the 2D "positioning" of the lidar (i.e., obtaining the distance between the obstacle and the AGV) and the 3D "semantics" of the binocular camera (i.e., obtaining the class of the obstacle). Specifically, different obstacle avoidance processing logics are designed for different classes of obstacles, so that the AGV can adopt appropriate obstacle avoidance strategies according to the class of the obstacle. The obstacle avoidance processing logic mainly can include the obstacle avoidance processing logics for pedestrians, vehicles, and shelves respectively.

[0160] Adopting the above AGV obstacle avoidance method combines the respective advantages of lidar-based obstacle avoidance and vision sensor-based obstacle avoidance. Through the fusion of lidar data and binocular camera data, more comprehensive environmental perception is achieved. This fusion not only improves the accuracy of obstacle detection, but also provides rich geometric and texture information, supporting more flexible obstacle avoidance strategies; through deep learning algorithms, the classification and recognition of obstacles can be realized, providing a dual obstacle avoidance function of "positioning + recognition". This semantic recognition ability enables the AGV to adopt targeted obstacle avoidance strategies according to the class of the obstacle, improving the intelligence level of the obstacle avoidance system and providing a more accurate, intelligent, and adaptable solution for AGV obstacle avoidance.

[0161] Based on the above AGV obstacle avoidance method, the embodiment of the present invention also provides an AGV obstacle avoidance system. As shown in Figure 4 , this system can include the following modules:

[0162] A preprocessing module 402, configured to obtain the original two-dimensional point cloud collected in real time by a lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by a binocular camera on the AGV, and preprocess the original two-dimensional point cloud and the original three-dimensional point cloud; wherein, the binocular camera uses an RGB camera and a depth camera.

[0163] A first processing module 404, configured to determine, based on the size information of each shelf leg, a first target point cloud in the preprocessed two-dimensional point cloud that corresponds to a target position and belongs to a first shelf leg area, and convert the non-first target point cloud in the preprocessed two-dimensional point cloud into a first point cloud in the vehicle body coordinate system corresponding to the AGV; wherein, the target position is the position where the AGV is located when the AGV moves to directly below the target shelf.

[0164] A second processing module 406, configured to convert the preprocessed three-dimensional point cloud into an initial three-dimensional point cloud in the vehicle body coordinate system, and delete a second target point cloud belonging to a target area in the initial three-dimensional point cloud based on ground information and the size information of the shelf leg, to obtain a second point cloud; wherein, the target area includes a second shelf leg area and a ground area.

[0165] An obstacle avoidance module 408, configured to perform obstacle detection on the first point cloud based on the motion state of the AGV, classify obstacles in the original RGB image and the second point cloud, and then determine an obstacle avoidance strategy for the AGV based on the obstacle detection result and the obstacle classification result.

[0166] By using the above AGV obstacle avoidance system, lidar data and binocular camera data can be combined for obstacle detection and classification, which not only improves the accuracy of obstacle detection, but also provides the type of obstacle as semantic information, so that the AGV can perform targeted obstacle avoidance according to the type of obstacle, improving the accuracy and flexibility of AGV obstacle avoidance.

[0167] The above obstacle avoidance module 408 can also be used to: determine an obstacle avoidance area for the AGV based on the motion state of the AGV and divide the obstacle avoidance area into multiple sub-areas; divide the first point cloud into sub-point clouds corresponding to the multiple sub-areas, and perform obstacle detection on each sub-point cloud.

[0168] The above motion states may include a straight-line motion state and a stationary rotation motion state; based on this, the above obstacle avoidance module 408 may further be configured to: if the AGV is in a straight-line motion state, determine the obstacle avoidance area as a rectangular obstacle avoidance area, and divide the rectangular obstacle avoidance area into a plurality of rectangular sub-areas; wherein, each rectangular sub-area corresponds to a corresponding driving range of the AGV; if the AGV is in a stationary rotation motion state, determine the obstacle avoidance area as a circular obstacle avoidance area, and divide the circular obstacle avoidance area into a plurality of fan-shaped sub-areas; wherein, each fan-shaped sub-area corresponds to a corresponding rotation angle range of the AGV.

[0169] The above obstacle detection results may include the distance between each detected obstacle and the AGV; based on this, the above obstacle avoidance module 408 may further be configured to: cluster each sub-point cloud into a corresponding cluster by using a preset clustering algorithm, and use the cluster with the number of points corresponding to each sub-point cloud less than a preset point number threshold as an effective cluster representing an obstacle; calculate the minimum distance between each effective cluster and the AGV, and use each obtained minimum distance as the distance between the corresponding obstacle and the AGV.

[0170] The above obstacle avoidance module 408 may further be configured to: extract features from the original RGB image and the second point cloud by using a preset feature extraction network to obtain a first feature corresponding to the original RGB image and a second feature corresponding to the second point cloud; perform classification prediction on the fused features by using a pre-trained convolutional neural network, and use the classification prediction result as the obstacle classification result; wherein, the classification prediction result includes the classification probability of the obstacle corresponding to each category.

[0171] The above first processing module 404 may further be configured to: determine the original coordinates of each shelf leg in the shelf coordinate system corresponding thereto based on the size information of each shelf leg, and convert the original coordinates of each shelf leg into first coordinates in the laser coordinate system corresponding to the lidar; wherein, the original coordinates include the four corner point coordinates and the center point coordinate of the corresponding shelf leg; perform angular domain constraint and spatial domain constraint on the first coordinates of each shelf leg to determine the angular range and spatial range of each shelf leg; when the AGV moves to directly below the target shelf, obtain the current two-dimensional point cloud corresponding to the current position of the AGV in the preprocessed two-dimensional point cloud, and use the two-dimensional points in the current two-dimensional point cloud that are both within the angular range and within the spatial range as target points to form the first target point cloud.

[0172] The above-mentioned preprocessing module 402 can also be used to: determine the distance value between each original two-dimensional point in the original two-dimensional point cloud and each of its adjacent points, and compare the distance value corresponding to each original two-dimensional point with a preset distance threshold to delete the isolated points in the original two-dimensional point cloud; wherein, the distance value between each isolated point and each of its adjacent points is greater than the preset distance threshold; perform voxel grid filtering on the original three-dimensional point cloud.

[0173] The above categories may include pedestrians, vehicles, and shelves; based on this, the above-mentioned obstacle avoidance module 408 can also be used to perform the following operations:

[0174] If the obstacle is a pedestrian, it is determined that the obstacle avoidance strategy of the AGV includes: when the distance between the pedestrian and the AGV is less than a preset first safety distance, controlling the AGV to decelerate and bypass the pedestrian; when planning the path, controlling the AGV to select a path far from the pedestrian to drive, or when the distance between the pedestrian and the AGV is less than the preset first safety distance, controlling the AGV to pause until the distance between the first obstacle and the AGV is not less than the preset first safety distance, and then controlling the AGV to continue driving; adjusting the driving speed of the AGV according to the movement state of the pedestrian so that the distance between the pedestrian and the AGV is greater than the preset first safety distance;

[0175] If the obstacle is a vehicle, it is determined that the obstacle avoidance strategy of the AGV includes: if the vehicle is in a stationary state, planning a first obstacle avoidance path based on the size and position of the vehicle and controlling the AGV to drive along the first obstacle avoidance path to bypass the vehicle; if the vehicle is in a moving state, predicting the movement trajectory of the vehicle and adjusting the current driving path of the AGV according to the movement trajectory; if the movement trajectory passes through a position point whose distance from the AGV is less than a preset second safety distance, controlling the AGV to decelerate so that the distance between each position point on the movement trajectory and the AGV is greater than the preset second safety distance; if the movement trajectory coincides with the current driving path of the AGV, controlling the driving speed of the AGV to be the same as the driving speed of the vehicle;

[0176] If the obstacle is a shelf, it is determined that the obstacle avoidance strategy of the AGV includes: planning a second obstacle avoidance path based on the size and position of the shelf and controlling the AGV to drive along the second obstacle avoidance path so that the distance between the shelf and the AGV is greater than a preset third safety distance; if the shelf is on the current driving path of the AGV, controlling the AGV to stop at the current position and re-plan the path, and then controlling the AGV to drive along the re-planned path so that the distance between the shelf and the AGV is greater than the preset third safety distance; optimizing the current driving path of the AGV to reduce the distance between the optimized path and the target position.

[0177] The AGV obstacle avoidance system provided by the embodiments of the present invention has the same implementation principle and technical effects as those of the foregoing embodiments of the AGV obstacle avoidance method. For the sake of brief description, for the parts not mentioned in the embodiments of the AGV obstacle avoidance system, reference may be made to the corresponding content in the foregoing embodiments of the AGV obstacle avoidance method.

[0178] The embodiments of the present invention also provide an electronic device. As Figure 5 shown, it is a schematic structural diagram of the electronic device. Among them, the electronic device includes a processor 51 and a memory 50. The memory 50 stores computer-executable instructions that can be executed by the processor 51, and the processor 51 executes the computer-executable instructions to implement the above AGV obstacle avoidance method.

[0179] In Figure 5 the illustrated embodiment, the electronic device further includes a bus 52 and a communication interface 53. Among them, the processor 51, the communication interface 53, and the memory 50 are connected through the bus 52.

[0180] Among them, the memory 50 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 53 (which may be wired or wireless), a communication connection is established between the system network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 52 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 52 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 5 only a bidirectional arrow is used in

[0181] The processor 51 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method can be completed by the integrated logic circuit of the hardware in the processor 51 or instructions in the form of software. The above-mentioned processor 51 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the AGV obstacle avoidance method disclosed in the embodiments of the present invention can be directly implemented by the execution of the hardware decoding processor, or by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor 51 reads the information in the memory and combines its hardware to complete the steps of the AGV obstacle avoidance method in the foregoing embodiments.

[0182] Unless otherwise specifically stated, the relative steps, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0183] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present invention essentially or the part that contributes to the prior art or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc that can store program codes.

[0184] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0185] Finally, it should be noted that the above-described embodiments are only specific embodiments of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments or easily conceive of changes, or make equivalent replacements for some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims

1. An AGV obstacle avoidance method, characterized in that, Including: Obtaining the original two-dimensional point cloud collected in real time by the lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by the binocular camera on the AGV, and preprocessing the original two-dimensional point cloud and the original three-dimensional point cloud; wherein, the binocular camera uses an RGB camera and a depth camera; Determining, based on the size information of each shelf leg, the first target point cloud in the preprocessed two-dimensional point cloud corresponding to the target position and belonging to the first shelf leg area, and converting the non-first target point cloud in the preprocessed two-dimensional point cloud into the first point cloud in the vehicle body coordinate system corresponding to the AGV; wherein, the target position is the position where the AGV is located when the AGV moves to directly below the target shelf; Converting the preprocessed three-dimensional point cloud into the initial three-dimensional point cloud in the vehicle body coordinate system, and deleting the second target point cloud belonging to the target area in the initial three-dimensional point cloud based on the ground information and the size information of the shelf leg to obtain the second point cloud; wherein, the target area includes the second shelf leg area and the ground area; Performing obstacle detection on the first point cloud based on the motion state of the AGV, classifying obstacles on the original RGB image and the second point cloud, and then determining the obstacle avoidance strategy of the AGV based on the obstacle detection result and the obstacle classification result.

2. The AGV obstacle avoidance method according to claim 1, characterized in that Performing obstacle detection on the first point cloud based on the motion state of the AGV, including: Determining the obstacle avoidance area of the AGV based on the motion state of the AGV and dividing the obstacle avoidance area into multiple sub-areas; Dividing the first point cloud into sub-point clouds corresponding to multiple sub-areas, and performing obstacle detection on each sub-point cloud.

3. The AGV obstacle avoidance method according to claim 2, wherein The motion state includes a straight-line motion state and a stationary rotation motion state; determining the obstacle avoidance area of the AGV based on the motion state of the AGV and dividing the obstacle avoidance area into multiple sub-areas, including: If the AGV is in the straight-line motion state, determining the obstacle avoidance area as a rectangular obstacle avoidance area and dividing the rectangular obstacle avoidance area into multiple rectangular sub-areas; wherein, each rectangular sub-area corresponds to the corresponding driving range of the AGV; If the AGV is in the stationary rotation motion state, determining the obstacle avoidance area as a circular obstacle avoidance area and dividing the circular obstacle avoidance area into multiple fan-shaped sub-areas; wherein, each fan-shaped sub-area corresponds to the corresponding rotation angle range of the AGV.

4. The AGV obstacle avoidance method according to claim 2, characterized in that, The obstacle detection result includes the distance between each detected obstacle and the AGV; performing obstacle detection on each sub-point cloud, including: Using a preset clustering algorithm to cluster each sub-point cloud into corresponding clusters, and taking the clusters with the number of points in each sub-point cloud less than the preset point number threshold as effective clusters representing obstacles; Calculating the minimum distance between each effective cluster and the AGV, and taking each obtained minimum distance as the distance between the corresponding obstacle and the AGV.

5. The AGV obstacle avoidance method according to claim 4, wherein Classifying obstacles on the original RGB image and the second point cloud, including: Use a preset feature extraction network to extract features from the original RGB image and the second point cloud, obtaining the first feature corresponding to the original RGB image and the second feature corresponding to the second point cloud; Use a pre-trained convolutional neural network to classify and predict the fused features, and take the classification and prediction result as the obstacle classification result; wherein, the classification and prediction result includes the classification probabilities of the obstacles corresponding to various categories.

6. The AGV obstacle avoidance method according to claim 1, wherein Determine the first target point cloud corresponding to the target position and belonging to the first shelf leg area in the preprocessed two-dimensional point cloud based on the size information of each shelf leg, including: Determine the original coordinates of each shelf leg in the shelf coordinate system corresponding thereto based on the size information of each shelf leg, and convert the original coordinates of each shelf leg into the first coordinates in the laser coordinate system corresponding to the lidar; wherein, the original coordinates include the four corner point coordinates and the center point coordinate of the corresponding shelf leg; Perform angular domain constraint and spatial domain constraint on the first coordinates of each shelf leg to determine the angular range and spatial range of each shelf leg; When the AGV moves to directly below the target shelf, obtain the current two-dimensional point cloud corresponding to the current position of the AGV in the preprocessed two-dimensional point cloud, and use the two-dimensional points in the current two-dimensional point cloud that are both within the angular range and within the spatial range as target points to form the first target point cloud.

7. The AGV obstacle avoidance method according to claim 1, wherein Preprocess the original two-dimensional point cloud and the original three-dimensional point cloud, including: Determine the distance value between each original two-dimensional point in the original two-dimensional point cloud and each of its adjacent points, and compare the distance value corresponding to each original two-dimensional point with a preset distance threshold to delete the isolated points in the original two-dimensional point cloud; wherein, the distance value between each isolated point and each of its adjacent points is greater than the preset distance threshold; Perform voxel grid filtering on the original three-dimensional point cloud.

8. The AGV obstacle avoidance method according to claim 5, wherein The categories include pedestrians, vehicles, and shelves; Determine the obstacle avoidance strategy of the AGV based on the obstacle detection result and the obstacle classification result, including: If the obstacle is a pedestrian, determine that the obstacle avoidance strategy of the AGV includes: when the distance between the pedestrian and the AGV is less than a preset first safety distance, control the AGV to decelerate and bypass the pedestrian; when planning the path, control the AGV to choose a path far from the pedestrian to drive, or when the distance between the pedestrian and the AGV is less than the preset first safety distance, control the AGV to pause until the distance between the first obstacle and the AGV is not less than the preset first safety distance and then control the AGV to continue driving; adjust the driving speed of the AGV according to the movement state of the pedestrian so that the distance between the pedestrian and the AGV is greater than the preset first safety distance; If the obstacle is a vehicle, determining the obstacle avoidance strategy of the AGV includes: if the vehicle is in a stationary state, planning a first obstacle avoidance path based on the size and position of the vehicle and controlling the AGV to drive along the first obstacle avoidance path to bypass the vehicle; if the vehicle is in a moving state, predicting the movement trajectory of the vehicle and adjusting the current driving path of the AGV according to the movement trajectory; if the movement trajectory passes through a position point with a distance less than a preset second safety distance from the AGV, controlling the AGV to decelerate so that the distance between each position point on the movement trajectory and the AGV is greater than the preset second safety distance; if the movement trajectory coincides with the current driving path of the AGV, controlling the driving speed of the AGV to be the same as the driving speed of the vehicle. If the obstacle is a shelf, determining the obstacle avoidance strategy of the AGV includes: planning a second obstacle avoidance path based on the size and position of the shelf and controlling the AGV to drive along the second obstacle avoidance path so that the distance between the shelf and the AGV is greater than a preset third safety distance; if the shelf is on the current driving path of the AGV, controlling the AGV to stop at the current position and re-plan the path, and then controlling the AGV to drive along the re-planned path so that the distance between the shelf and the AGV is greater than the preset third safety distance; optimizing the current driving path of the AGV to reduce the distance between the optimized path and the target position.

9. An AGV obstacle avoidance system, characterized in that, Including: An acquisition module, configured to acquire the original two-dimensional point cloud collected in real time by the lidar on the AGV, the original three-dimensional point cloud and the original RGB image collected in real time by the binocular camera on the AGV, and preprocess the original two-dimensional point cloud and the original three-dimensional point cloud; wherein, the binocular camera uses an RGB camera and a depth camera. A first processing module, configured to determine, based on the size information of each shelf leg, a first target point cloud in the preprocessed two-dimensional point cloud that corresponds to the target position and belongs to the first shelf leg area, and convert the non-first target point cloud in the preprocessed two-dimensional point cloud into a first point cloud in the vehicle body coordinate system corresponding to the AGV; wherein, the target position is the position where the AGV is located when the AGV moves to directly below the target shelf. A second processing module, configured to convert the preprocessed three-dimensional point cloud into an initial three-dimensional point cloud in the vehicle body coordinate system, and delete the second target point cloud belonging to the target area in the initial three-dimensional point cloud based on the ground information and the size information of the shelf leg to obtain a second point cloud; wherein, the target area includes a second shelf leg area and a ground area. An obstacle avoidance module, configured to perform obstacle detection on the first point cloud based on the motion state of the AGV, classify obstacles on the original RGB image and the second point cloud, and then determine the obstacle avoidance strategy of the AGV based on the obstacle detection result and the obstacle classification result.

10. An electronic device, characterized in that, It includes a processor and a memory, and the memory stores computer-executable instructions that can be executed by the processor. The processor executes the computer-executable instructions to implement the AGV obstacle avoidance method according to any one of claims 1 to 8.

Images