Intelligent vehicle obstacle avoidance method, system and electronic device

Abstract

The application provides a kind of intelligent vehicle obstacle avoidance method, system and electronic equipment, the method comprises: obtaining the color image and depth image of the scene where target intelligent vehicle is located;The spatial conversion is carried out to depth image, to obtain first point cloud data, according to the spatial conversion relationship between depth camera coordinate system and color camera coordinate system, the spatial conversion is carried out to first point cloud data, to obtain second point cloud data;Obstacle detection is carried out to color image, to obtain obstacle region of interest;According to the second point cloud data of obstacle region of interest processing, to obtain first obstacle point cloud data;According to the distance information between obstacle and target intelligent vehicle obtained from first obstacle point cloud data;According to the distance information between obstacle and target intelligent vehicle and preset hierarchical obstacle avoidance strategy, control target intelligent vehicle executes corresponding obstacle avoidance action.The application has the advantages of high real-time detection precision and strong obstacle avoidance ability.

Description

Technical Field

[0001] This invention relates to the field of visual automatic obstacle avoidance technology, and in particular to an intelligent vehicle obstacle avoidance method, system and electronic device. Background Technology

[0002] In recent years, with the rapid development of logistics and automated warehousing technologies, program-controlled unmanned forklifts have become a research and application hotspot in the fields of cargo handling and warehouse management. Among them, environmental perception capability, as a key technology for achieving safe operation, has made intelligent forklift obstacle avoidance systems a focal point of industry attention.

[0003] In logistics and automated warehousing, common obstacle avoidance solutions for intelligent forklifts include: installing two obstacle avoidance radars at the bottom of the forklift to detect non-animal obstacles on the ground to the left and right of the forklift; installing a single-line lidar on the top of the forklift to detect non-animal obstacles on the top plane, but large areas on the front and back of the forklift are not involved in the obstacle avoidance scheme, so it cannot achieve a certain degree of obstacle avoidance effect when unexpected protruding non-animal obstacles appear in front of or behind the intelligent forklift while it is working normally; installing multi-line lidar on the top of the forklift, but the amount of radar point cloud data is huge, the requirements for hardware resources are relatively high, it cannot be adapted to low-end and mid-range industrial control computers, it is difficult to achieve simultaneous obstacle avoidance of multiple non-animal obstacles with limited resources, and the obstacle avoidance effect for smaller non-animal obstacles is poor, and the cost is high.

[0004] Therefore, there is an urgent need to develop a high-precision, high-real-time performance, and low-cost multi-mode visual obstacle avoidance method for intelligent forklifts.

[0005] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent vehicle obstacle avoidance method, system and electronic device, which has the advantages of high real-time detection accuracy and strong obstacle avoidance capability.

[0007] To achieve the above objectives, the present invention provides an intelligent vehicle obstacle avoidance method, comprising:

[0008] Acquire color and depth images of the scene where the target intelligent vehicle is located;

[0009] The depth image is spatially transformed to obtain the first point cloud data in the depth camera coordinate system. Based on the pre-calibrated spatial transformation relationship between the depth camera coordinate system and the color camera coordinate system, the first point cloud data is spatially transformed to obtain the second point cloud data in the color camera coordinate system.

[0010] Obstacle detection is performed on the color image to obtain at least one region of interest for the obstacle;

[0011] The second point cloud data is processed according to the region of interest of the obstacle to obtain the corresponding first obstacle point cloud data;

[0012] Based on the first obstacle point cloud data, obtain the distance information between the corresponding obstacle and the target intelligent vehicle;

[0013] Based on the distance information between the obstacle and the target intelligent vehicle and the preset hierarchical obstacle avoidance strategy, the target intelligent vehicle is controlled to perform the corresponding obstacle avoidance action.

[0014] Optionally, obstacle detection is performed on the color image, including:

[0015] Animal-type obstacles and non-animal-type obstacles are detected separately in color images. Animal-type obstacles include humans and other animals besides humans.

[0016] Optionally, based on the distance information between the obstacle and the target intelligent vehicle and a preset graded obstacle avoidance strategy, the target intelligent vehicle is controlled to perform corresponding obstacle avoidance actions, including:

[0017] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the target intelligent vehicle and the nearest non-animal-shaped obstacle is less than or equal to the first preset distance, then control the target intelligent vehicle to perform a preset prohibited area action.

[0018] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle are both greater than the first preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the second preset distance, then control the target intelligent vehicle to perform the preset warning zone action.

[0019] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform the preset safe zone action.

[0020] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a third preset distance, then the target intelligent vehicle is controlled to perform a preset special safe zone action.

[0021] Among them, the first preset distance is less than the second preset distance, and the second preset distance is less than the third preset distance.

[0022] Optionally, control the target intelligent vehicle to perform actions within a preset prohibited area, including:

[0023] Control the target intelligent vehicle to perform emergency stop and alarm actions.

[0024] Optionally, if the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a first preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to a second preset distance, then the target intelligent vehicle is controlled to perform a preset warning zone action, including:

[0025] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the second preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0026] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than a first preset distance and less than or equal to a second preset distance, then the target intelligent vehicle is controlled to perform an emergency stop.

[0027] Optionally, if the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to a third preset distance, then the target intelligent vehicle is controlled to perform a preset safe zone action, including:

[0028] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the third preset distance, then the global path of the target intelligent vehicle will be replanned, and the target intelligent vehicle will be controlled to maintain the preset operating speed.

[0029] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, then the target intelligent vehicle will be controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0030] Optionally, control the target intelligent vehicle to perform actions within a preset safe zone, including:

[0031] Control the target intelligent vehicle to maintain the preset operating speed.

[0032] Optionally, animal-type obstacles and non-animal-type obstacles are detected separately in the color image, including:

[0033] Both the pre-trained animal obstacle detection model and the non-animal obstacle detection model are converted into RKNN format, and pruning and quantization operations are performed on the animal obstacle detection model and the non-animal obstacle detection model during the conversion process.

[0034] Animal and non-animal obstacle detection models in RKNN format are run on the NPU to detect animal and non-animal obstacles in color images, respectively.

[0035] Optionally, both the animal-type obstacle detection model and the non-animal-type obstacle detection model are YOLOv8s-seg models.

[0036] Optionally, the second point cloud data is processed according to the region of interest of the obstacle to obtain the corresponding first obstacle point cloud data, including:

[0037] The second point cloud data is cropped based on the region of interest of the obstacle to obtain the initial obstacle point cloud data;

[0038] The initial obstacle point cloud data is subjected to downsampling, outlier removal, and Euclidean clustering operations in sequence to obtain the first obstacle point cloud data.

[0039] Optionally, distance information between the corresponding obstacle and the target intelligent vehicle can be obtained based on the first obstacle point cloud data, including:

[0040] Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first obstacle point cloud data is spatially transformed to obtain the second obstacle point cloud data in the target intelligent vehicle base coordinate system.

[0041] Principal component analysis and oriented bounding box processing are performed sequentially on the point cloud data of the second obstacle to obtain the corresponding oriented bounding box;

[0042] Based on the Z-axis coordinate of the center point of the directional bounding box, the distance information between the corresponding obstacle and the target intelligent vehicle is obtained.

[0043] Optionally, the depth image is spatially transformed to obtain the first point cloud data in the depth camera coordinate system, including:

[0044] Integer multiplication and displacement operation strategies are used to spatially transform the depth image to obtain the first point cloud data in the depth camera coordinate system.

[0045] To achieve the above objectives, the present invention also provides an intelligent vehicle obstacle avoidance system, comprising: an image acquisition module configured to acquire a color image and a depth image of the scene where the target intelligent vehicle is located; a conversion module configured to perform spatial conversion on the depth image to obtain first point cloud data in a depth camera coordinate system, and to perform spatial conversion on the first point cloud data according to a pre-calibrated spatial conversion relationship between the depth camera coordinate system and the color camera coordinate system to obtain second point cloud data in the color camera coordinate system; an obstacle detection module configured to detect obstacles in the color image to obtain at least one obstacle region of interest, and to process the second point cloud data according to the obstacle region of interest to obtain corresponding first obstacle point cloud data, and to obtain distance information between the corresponding obstacle and the target intelligent vehicle according to the first obstacle point cloud data; and a hierarchical obstacle avoidance module configured to control the target intelligent vehicle to perform corresponding obstacle avoidance actions according to the distance information between the obstacle and the target intelligent vehicle and a preset hierarchical obstacle avoidance strategy.

[0046] Optionally, the obstacle detection module includes: an animal obstacle detection unit configured to detect animal obstacles in the color image, including humans and other animals besides humans; and a non-animal obstacle detection unit configured to detect non-animal obstacles in the color image.

[0047] Optionally, the graded obstacle avoidance module can be configured as follows:

[0048] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the target intelligent vehicle and the nearest non-animal-shaped obstacle is less than or equal to the first preset distance, then control the target intelligent vehicle to perform a preset prohibited area action.

[0049] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle are both greater than the first preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the second preset distance, then control the target intelligent vehicle to perform the preset warning zone action.

[0050] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform the preset safe zone action.

[0051] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a third preset distance, then the target intelligent vehicle is controlled to perform a preset special safe zone action.

[0052] Among them, the first preset distance is less than the second preset distance, and the second preset distance is less than the third preset distance.

[0053] Optionally, control the target intelligent vehicle to perform actions within a preset prohibited area, including:

[0054] Control the target intelligent vehicle to perform emergency stop and alarm actions.

[0055] Optionally, the graded obstacle avoidance module can be configured as follows:

[0056] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the second preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0057] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than a first preset distance and less than or equal to a second preset distance, then the target intelligent vehicle is controlled to perform an emergency stop.

[0058] Optionally, the graded obstacle avoidance module can be configured as follows:

[0059] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the third preset distance, then the global path of the target intelligent vehicle will be replanned, and the target intelligent vehicle will be controlled to maintain the preset operating speed.

[0060] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, then the target intelligent vehicle will be controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0061] Optionally, control the target intelligent vehicle to perform actions within a preset safe zone, including:

[0062] Control the target intelligent vehicle to maintain the preset operating speed.

[0063] Optionally, the animal obstacle detection unit is configured to: convert a pre-trained animal obstacle detection model into RKNN format, perform pruning and quantization operations on the animal obstacle detection model during the conversion process, and run the RKNN format animal obstacle detection model on the NPU to detect animal obstacles in the color image; the non-animal obstacle detection unit is configured to: convert a pre-trained non-animal obstacle detection model into RKNN format, perform pruning and quantization operations on the non-animal obstacle detection model during the conversion process, and run the RKNN format non-animal obstacle detection model on the NPU to detect non-animal obstacles in the color image.

[0064] Optionally, both the animal-type obstacle detection model and the non-animal-type obstacle detection model are YOLOv8s-seg models.

[0065] Optionally, the obstacle conversion module can be configured as follows:

[0066] The second point cloud data is cropped based on the region of interest of the obstacle to obtain the initial obstacle point cloud data;

[0067] The initial obstacle point cloud data is subjected to downsampling, outlier removal, and Euclidean clustering operations in sequence to obtain the first obstacle point cloud data.

[0068] Optionally, the obstacle detection module can be configured as follows:

[0069] Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first obstacle point cloud data is spatially transformed to obtain the second obstacle point cloud data in the target intelligent vehicle base coordinate system.

[0070] Principal component analysis and oriented bounding box processing are performed sequentially on the point cloud data of the second obstacle to obtain the corresponding oriented bounding box;

[0071] Based on the Z-axis coordinate of the center point of the directional bounding box, the distance information between the corresponding obstacle and the target intelligent vehicle is obtained.

[0072] Optionally, the conversion module is configured to perform spatial transformation on the depth image using integer multiplication and displacement operation strategies to obtain the first point cloud data in the depth camera coordinate system.

[0073] To achieve the above objectives, the present invention also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the intelligent vehicle obstacle avoidance method described above is implemented.

[0074] Compared with existing technologies, the intelligent vehicle obstacle avoidance method, system, and electronic device provided by this invention have the following beneficial effects:

[0075] The intelligent vehicle obstacle avoidance method provided by this invention first acquires color images (e.g., RGB images) and depth images of the scene where the target intelligent vehicle (e.g., an unmanned forklift) is located. Then, obstacle point cloud data is obtained based on the color and depth images. Next, distance information between the obstacle and the target intelligent vehicle is obtained based on the obstacle point cloud data. Finally, graded obstacle avoidance is performed based on the distance information between the obstacle and the target intelligent vehicle. This allows for the automatic adoption of different obstacle avoidance strategies for obstacles at different distances, effectively reducing human intervention and improving the safety and intelligence of intelligent vehicles (e.g., unmanned forklifts) during operation. Furthermore, the intelligent vehicle obstacle avoidance method provided by this invention has high real-time performance, achieving a detection efficiency of 15 frames per second (i.e., a detection efficiency of 15fps), ensuring millisecond-level response latency and meeting high real-time obstacle avoidance requirements. In addition, the intelligent vehicle obstacle avoidance method provided by this invention has high detection sensitivity: it can effectively identify small, irregularly shaped obstacles (minimum cross-sectional area ≥ 5cm²) within 3 meters. 2 Stable detection of irregularly shaped obstacles of medium volume within 5 meters (minimum cross-sectional area ≥ 30cm²) 2 Reliably detect large-volume obstacles (minimum cross-sectional area ≥ 1m²) within 7 meters. 2Meanwhile, the intelligent vehicle obstacle avoidance method provided by this invention has strong obstacle avoidance capabilities, achieving an effective obstacle avoidance success rate of over 98%, and a dynamic response time of less than 70 milliseconds.

[0076] Since the intelligent vehicle obstacle avoidance system and electronic device provided by this invention belong to the same inventive concept as the intelligent vehicle obstacle avoidance method provided by this invention, the intelligent vehicle obstacle avoidance system and electronic device provided by this invention have at least all the beneficial effects of the intelligent vehicle obstacle avoidance method provided by this invention. For details, please refer to the relevant description above. Therefore, the beneficial effects of the intelligent vehicle obstacle avoidance system and electronic device provided by this invention will not be elaborated here. Attached Figure Description

[0077] Figure 1 A flowchart of an intelligent vehicle obstacle avoidance method provided in one embodiment of the present invention.

[0078] Figure 2 This is a schematic diagram of a preset hierarchical obstacle avoidance strategy provided in one embodiment of the present invention.

[0079] Figure 3 This is a block diagram of an intelligent vehicle obstacle avoidance system provided in one embodiment of the present invention.

[0080] Figure 4 This is a block diagram of an electronic device provided according to an embodiment of the present invention.

[0081] The reference numerals in the attached drawings are explained as follows: Image acquisition module - 110; Conversion module - 120; Obstacle detection module - 130; Animal-type obstacle detection unit - 131; Non-animal-type obstacle detection unit - 132; Hierarchical obstacle avoidance module - 140; Processor - 210; Communication interface - 220; Memory - 230; Communication bus - 240. Detailed Implementation

[0082] The intelligent vehicle obstacle avoidance method, system, and electronic device proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Please refer to the accompanying drawings for the objectives, features, and advantages of this invention to make them more apparent. It should be noted that similar reference numerals and letters are used to denote similar items in this specification; therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings. Furthermore, if the method herein comprises a series of steps, the order of these steps presented herein is not necessarily the only possible order in which these steps can be performed, and some steps may be omitted and / or other steps not described herein may be added to the method.

[0083] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element. The singular forms “a,” “an,” and “the” include plural objects. The term “or” is generally used to mean “and / or,” the term “several” is generally used to mean “at least one,” and the term “at least two” is generally used to mean “two or more.” Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0084] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] The core idea of ​​this invention is to provide an intelligent vehicle obstacle avoidance method, system, and electronic device, which has the advantages of high real-time detection accuracy and strong obstacle avoidance capability.

[0086] It should be noted that the intelligent vehicle obstacle avoidance method provided by this invention can be applied to the intelligent vehicle obstacle avoidance system provided by this invention. The intelligent vehicle obstacle avoidance system can be configured on an electronic device, wherein the electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, a tablet computer, or other hardware device with various operating systems. It should also be noted that the "intelligent vehicle" referred to in this invention can be, but is not limited to, an unmanned forklift used for cargo handling and warehouse management.

[0087] To achieve the above-mentioned goals, this invention provides an intelligent vehicle obstacle avoidance method, please refer to... Figure 1 This is a flowchart of an intelligent vehicle obstacle avoidance method provided in one embodiment of the present invention. Figure 1 As shown, the obstacle avoidance method for intelligent vehicles includes the following steps:

[0088] Step S100: Obtain color and depth images of the scene where the target intelligent vehicle is located;

[0089] Step S200: Perform spatial transformation on the depth image to obtain the first point cloud data in the depth camera coordinate system, and perform spatial transformation on the first point cloud data according to the pre-calibrated spatial transformation relationship between the depth camera coordinate system and the color camera coordinate system to obtain the second point cloud data in the color camera coordinate system.

[0090] Step S300: Detect obstacles in the color image to obtain at least one region of interest for an obstacle;

[0091] Step S400: Process the second point cloud data according to the region of interest of the obstacle to obtain the corresponding first obstacle point cloud data;

[0092] Step S500: Obtain the distance information between the corresponding obstacle and the target intelligent vehicle based on the first obstacle point cloud data;

[0093] Step S600: Based on the distance information between the obstacle and the target intelligent vehicle and the preset hierarchical obstacle avoidance strategy, control the target intelligent vehicle to perform the corresponding obstacle avoidance action.

[0094] The intelligent vehicle obstacle avoidance method provided by this invention first acquires color images (e.g., RGB images) and depth images of the scene where the target intelligent vehicle (e.g., an unmanned forklift) is located. Then, obstacle point cloud data is obtained based on the color and depth images. Next, distance information between the obstacle and the target intelligent vehicle is obtained based on the obstacle point cloud data. Finally, graded obstacle avoidance is performed based on the distance information between the obstacle and the target intelligent vehicle. This allows for the automatic adoption of different obstacle avoidance strategies for obstacles at different distances, effectively reducing human intervention and improving the safety and intelligence of intelligent vehicles (e.g., unmanned forklifts) during operation. Furthermore, the intelligent vehicle obstacle avoidance method provided by this invention has high real-time performance, achieving a detection efficiency of 15 frames per second (i.e., a detection efficiency of 15fps), ensuring millisecond-level response latency and meeting high real-time obstacle avoidance requirements. In addition, the intelligent vehicle obstacle avoidance method provided by this invention has high detection sensitivity: it can effectively identify small, irregularly shaped obstacles (minimum cross-sectional area ≥ 5cm²) within 3 meters. 2 Stable detection of irregularly shaped obstacles of medium volume within 5 meters (minimum cross-sectional area ≥ 30cm²) 2Reliably detect large-volume obstacles (minimum cross-sectional area ≥ 1m²) within 7 meters. 2 Meanwhile, the intelligent vehicle obstacle avoidance method provided by this invention has strong obstacle avoidance capabilities, achieving an effective obstacle avoidance success rate of over 98%, and a dynamic response time of less than 70 milliseconds.

[0095] Specifically, an industrial camera equipped with a color camera (e.g., an RGB camera) and a depth camera (using an RK3588 chip, supporting IEEE 802.3at PoE+ power supply and gigabit Ethernet data transmission) can be installed at both the front and rear of the target intelligent vehicle's roof. The first industrial camera at the front of the target intelligent vehicle's roof can simultaneously acquire color and depth images of the scene in front of the vehicle, while the second industrial camera at the rear of the target intelligent vehicle's roof can simultaneously acquire color and depth images of the scene behind the vehicle. Furthermore, during the target intelligent vehicle's forward movement, obstacle avoidance can be performed based on the color and depth images of the scene in front of the vehicle acquired by the first industrial camera; similarly, during the target intelligent vehicle's backward movement, obstacle avoidance can be performed based on the color and depth images of the scene behind the vehicle acquired by the second industrial camera.

[0096] Furthermore, for each industrial camera (including the first industrial camera and the second industrial camera), the color camera and depth camera in the camera need to be calibrated for internal and external parameters before use.

[0097] Intrinsic calibration of a camera refers to the process of determining the camera's internal optical and geometric parameters, mainly including key data such as focal length, distortion parameters, optical axis center coordinates, and pixel size. Extrinsic calibration allows the acquisition of the spatial transformation relationship between the color camera and the depth camera. Furthermore, a calibration board of a certain size (e.g., a 10×7 checkerboard) can be used, employing a nonlinear least squares method (such as the Levenberg-Marquardt algorithm) to minimize reprojection errors. Through iterative optimization, the intrinsic parameters and distortion coefficients of the camera (color camera / depth camera), as well as the spatial transformation relationship between the color camera and the depth camera, can be solved.

[0098] The specific internal parameter calibration process includes:

[0099] (1) Prepare a calibration board, which is usually a checkerboard array of known size. The position of each corner point on the calibration board is known and can be used as a reference point.

[0100] (2) Multiple color and depth images containing the calibration plate are taken simultaneously from different angles using a color camera and a depth camera.

[0101] (3) For each color image / depth image, use an image processing algorithm (such as OpenCV's cv2.findChessboardCorners) to detect the corner points of the calibration board in the color image / depth image and record their pixel coordinates. Furthermore, in order to improve accuracy, subpixel-level optimization methods (such as cv2.cornerSubPix) can be used to further optimize the position of the corner points.

[0102] (4) Based on the known size and number of corner points of the calibration plate, prepare the 3D world coordinates (actual coordinates) and the corresponding 2D image points (pixel coordinates) for each color image / depth image.

[0103] (5) Calculate the intrinsic parameter matrix and distortion coefficients of the color camera / depth camera using a calibration algorithm (such as cv2.calibrateCamera). This step requires inputting 3D-2D point pairs from multiple color / depth images into the calibration algorithm for global optimization.

[0104] (6) By performing distortion correction and reprojection error analysis on the color / depth images, the accuracy of the calibration results can be verified. This operation can eliminate linear distortion caused by lens optical distortion, making the image more consistent with the geometric relationships of the real scene. Furthermore, some new color / depth images can be used for verification to check whether the calibration results meet expectations. Even further, if the reprojection error is less than 0.3 pixels, the calibration results can be determined to meet expectations.

[0105] Furthermore, for color / depth images, the 3D world coordinates of these color / depth images can be mapped to 2D image points on the image plane using the intrinsic and extrinsic parameters (including rotation and translation matrices) of the color / depth camera. That is:

[0106]

[0107] in, Represents points in a 2D image. K represents the 3D world coordinates, and K represents the intrinsic parameter matrix. R represents the rotation matrix, and T represents the translation matrix.

[0108] Therefore, the extrinsic parameter matrices of the color camera and depth camera can be solved. Furthermore, based on the extrinsic parameter matrices of the color camera and depth camera, the spatial transformation relationship between the depth camera coordinate system and the color camera coordinate system can be solved. Specifically, for details on how to solve for the spatial transformation relationship between the depth camera coordinate system and the color camera coordinate system based on the extrinsic parameter matrices of the color camera and depth camera, please refer to relevant materials well-known to those skilled in the art for an appropriate understanding; further explanation is not provided here.

[0109] It should be noted that, as those skilled in the art will understand, the calibration board can also be a dot array. For details on how to use a dot array calibration board for internal and external parameter calibration, please refer to the relevant content above for an adaptive understanding, and will not be repeated here.

[0110] Furthermore, the depth image can be spatially transformed based on the pre-calibrated intrinsic parameter matrix of the depth camera coordinate system to obtain the first point cloud data in the depth camera coordinate system. The specific calculation formula is as follows:

[0111]

[0112] Where (X,Y,Z) are the coordinates of the points in the depth camera coordinate system, (x,y) are the coordinates of the points in the depth image coordinate system, and z is the corresponding depth value.

[0113] In some exemplary implementations, a spatial transformation is performed on the depth image to obtain first point cloud data in the depth camera coordinate system, including:

[0114] Integer multiplication and displacement operation strategies are used to spatially transform the depth image to obtain the first point cloud data in the depth camera coordinate system.

[0115] Since the depth values ​​are output in meters, this results in floating-point operations. Combined with the intensive operations involving a large number of pixels, this leads to significant resource consumption. This invention transforms floating-point operations into integer multiplication and bitwise operations, which not only effectively improves conversion efficiency and saves CPU resources (reducing CPU usage from 85% to 45%), thus solving the problem of excessive resource consumption, but also has little impact on the frame rate of image output. This provides a stable and efficient data source for subsequent input data.

[0116] Furthermore, a PCL (Point Cloud Library) can be used to perform spatial transformation on the depth image. Even further, since noise and missing values ​​are generated during the depth image to point cloud conversion process, the first point cloud data obtained from the depth image conversion can be filtered and subjected to nearest neighbor difference processing before spatial transformation is performed on the processed first point cloud data to obtain the second point cloud data in the camera coordinate system.

[0117] In some exemplary implementations, obstacle detection on a color image includes:

[0118] Animal-type obstacles and non-animal-type obstacles are detected separately in color images. Animal-type obstacles include humans and other animals besides humans.

[0119] Therefore, by detecting animal-type obstacles and non-animal-type obstacles separately in the color images of the scene where the target intelligent vehicle is located, not only can the detection accuracy of obstacles be further improved, but also it is convenient to adopt different obstacle avoidance strategies based on the distance information between animal-type obstacles and the target intelligent vehicle and the distance information between non-animal-type obstacles and the target intelligent vehicle. This can further improve the safety of intelligent vehicles (such as unmanned forklifts) during operation, while ensuring the safety of people and other animals.

[0120] It should be noted that, as those skilled in the art will understand, when the scene in which the target intelligent vehicle is located includes multiple animal-shaped obstacles, multiple regions of interest (ROIs) for animal-shaped obstacles can be obtained by detecting these obstacles in the color image. For each ROI, corresponding first animal-shaped obstacle point cloud data is obtained. Based on each first animal-shaped obstacle point cloud data, the distance between the corresponding animal-shaped obstacle and the target intelligent vehicle can be obtained. Similarly, when the scene in which the target intelligent vehicle is located includes multiple non-animal-shaped obstacles, multiple regions of interest for non-animal-shaped obstacles can be obtained by detecting these obstacles in the color image. For each non-animal-shaped obstacle ROI, corresponding first non-animal-shaped obstacle point cloud data is obtained. Based on each first non-animal-shaped obstacle point cloud data, the distance between the corresponding non-animal-shaped obstacle and the target intelligent vehicle can be obtained.

[0121] In some exemplary implementations, animal-type obstacles and non-animal-type obstacles are detected separately in the color image, including:

[0122] Both the pre-trained animal obstacle detection model and the non-animal obstacle detection model are converted into RKNN format, and pruning and quantization operations are performed on the animal obstacle detection model and the non-animal obstacle detection model during the conversion process.

[0123] Animal and non-animal obstacle detection models in RKNN format are run on the NPU to detect animal and non-animal obstacles in color images, respectively.

[0124] Therefore, by running the RKNN format animal obstacle detection model and non-animal obstacle detection model on the NPU (Neural Processing Unit), the CPU (Central Processing Unit) resource usage can be reduced, thereby meeting real-time requirements and effectively improving the detection accuracy of both animal and non-animal obstacles.

[0125] Furthermore, the animal-type obstacle detection model and the non-animal-type obstacle detection model can be, but are not limited to, the YOLOv8s-seg model. For the specific structure of the YOLOv8s-seg model, please refer to relevant materials known to those skilled in the art for an adaptive understanding; further details will not be elaborated here. Taking the animal-type obstacle detection model and the non-animal-type obstacle detection model as examples using the YOLOv8s-seg model, with a single input color image of 960x540 resolution, the processing time using the CPU directly is 300ms (3.3fps), and with NPU acceleration, it is 60ms (16.6fps). Furthermore, the model can be enhanced with quantization, for example, enabling dynamic quantization during RKNN transformation can reduce INT8 accuracy loss while maintaining speed; retaining FP16 for the sensitive layer (segmentation head) and INT8 for the remaining layers can balance speed and accuracy. Furthermore, a multi-core NPU load balancing strategy can be used (that is, the computational tasks of the model are dynamically and reasonably distributed to multiple processing cores within the NPU so that these cores can work at full load as much as possible at the same time). This can automatically adjust the computational intensity according to hardware resources, ensuring that a stable frame rate (≥25fps) can still be maintained in complex scenes, and reducing latency by about 10%-20% (latency reduced from 60ms to 50ms).

[0126] Taking an animal obstacle detection model as an example, the acquired color images containing animal obstacles (such as people) can first be finely annotated to accurately extract the animal obstacle targets from the color images. Then, the model is trained based on the YOLOv8s-seg model, generating a PyTorch format model file (.pt). To achieve efficient deployment of the animal obstacle detection model on an NPU, the trained model needs to be converted to RKNN format. This conversion process mainly includes: parsing the original model structure and parameters, converting to an ONNX intermediate representation, performing FP32 to INT8 / INT16 quantization to adapt to NPU hardware characteristics, performing equivalent replacements for unsupported operators, implementing performance optimizations such as graph optimization and operator fusion, and finally generating a dedicated RKNN binary model file. Therefore, through advanced model compression and acceleration techniques, millisecond-level inference speeds (≤60ms) can be achieved while ensuring detection confidence (≥80%). By employing optimization strategies such as mixed-precision quantization, operator fusion, and memory reuse, the computational complexity and memory footprint of the model can be significantly reduced, keeping the peak power consumption of the YOLOv8s-seg model on embedded platforms below 5W, perfectly meeting the real-time requirements of mobile devices. Furthermore, when converting the model, it is necessary to check RKNN's support for PyTorch operators to ensure that the model's input and output are consistent with expectations and to confirm the target platform. It should be noted that more information regarding model training can be found in relevant materials known to those skilled in the art and will not be elaborated upon here.

[0127] In some exemplary implementations, the second point cloud data is processed according to the region of interest of the obstacle to obtain the corresponding first obstacle point cloud data, including:

[0128] The second point cloud data is cropped based on the region of interest of the obstacle to obtain the initial obstacle point cloud data;

[0129] The initial obstacle point cloud data is subjected to downsampling, outlier removal, and Euclidean clustering operations in sequence to obtain the first obstacle point cloud data.

[0130] Therefore, by cropping the second point cloud data based on the region of interest of the obstacle, the location of the obstacle can be directly locked, which helps to further improve the obstacle detection efficiency and meet the low latency requirements of real-time obstacle avoidance. By downsampling the initial obstacle point cloud data obtained by cropping, the computational load can be reduced, which helps to further improve the obstacle detection efficiency. By removing outliers from the downsampled initial obstacle point cloud data, noise can be effectively eliminated, which further improves the obstacle detection accuracy. By performing Euclidean clustering on the initial obstacle point cloud data after removing outliers, point clouds belonging to the same obstacle can be segmented, which can lay a good foundation for obtaining the distance information between the obstacle and the target intelligent vehicle.

[0131] Specifically, based on the region of interest of the obstacle and combined with the intrinsic parameter matrix of the pre-calibrated color camera, the initial obstacle point cloud data corresponding to the region of interest of the obstacle can be cropped from the second point cloud data.

[0132] It should be noted that, for each region of interest (ROI) of an animal-shaped obstacle, the first animal-shaped obstacle point cloud data corresponding to that ROI is obtained through the following steps:

[0133] The second point cloud data is cropped based on the region of interest of the animal-shaped obstacle to obtain the initial animal-shaped obstacle point cloud data;

[0134] The initial animal-shaped obstacle point cloud data is sequentially downsampled, outlier removed, and Euclidean clustered to obtain the corresponding first animal-shaped obstacle point cloud data.

[0135] For each region of interest (ROI) of a non-animal obstacle, the following steps are taken to obtain the first non-animal obstacle point cloud data corresponding to that ROI:

[0136] The second point cloud data is cropped based on the region of interest of the non-animal obstacle to obtain the initial non-animal obstacle point cloud data;

[0137] The initial non-animal obstacle point cloud data is sequentially downsampled, outlier removed, and Euclidean clustered to obtain the corresponding first non-animal obstacle point cloud data.

[0138] In some exemplary implementations, distance information between the corresponding obstacle and the target intelligent vehicle is obtained based on the first obstacle point cloud data, including:

[0139] Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first obstacle point cloud data is spatially transformed to obtain the second obstacle point cloud data in the target intelligent vehicle base coordinate system.

[0140] Principal component analysis and oriented bounding box processing are performed sequentially on the point cloud data of the second obstacle to obtain the corresponding oriented bounding box;

[0141] Based on the Z-axis coordinate of the center point of the directional bounding box, the distance information between the corresponding obstacle and the target intelligent vehicle is obtained.

[0142] Therefore, by first transforming the first obstacle point cloud data from the color camera coordinate system to the target intelligent vehicle base coordinate system to obtain the second obstacle point cloud data, and then performing principal component analysis and oriented bounding box processing on the second obstacle point cloud data, the distance information between the obstacle and the target intelligent vehicle can be obtained quickly and accurately.

[0143] It should be noted that the specific details regarding how to calibrate the spatial transformation relationship between the color camera coordinate system and the target intelligent vehicle base coordinate system can be adapted by referring to relevant content known to those skilled in the art, and will not be elaborated here.

[0144] It should be noted that, for each first animal-shaped obstacle point cloud data, the distance information between the corresponding animal-shaped obstacle and the target intelligent vehicle is obtained through the following steps:

[0145] Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first animal-shaped obstacle point cloud data is spatially transformed to obtain the second animal-shaped obstacle point cloud data in the target intelligent vehicle base coordinate system.

[0146] Principal component analysis and oriented bounding box processing were performed sequentially on the point cloud data of the second animal-type obstacle to obtain the first oriented bounding box;

[0147] Based on the Z-axis coordinates of the center point of the first directional bounding box, the distance information between the corresponding animal-shaped obstacle and the target intelligent vehicle is obtained.

[0148] Therefore, by first transforming the first animal-shaped obstacle point cloud data from the color camera coordinate system to the target intelligent vehicle base coordinate system to obtain the second animal-shaped obstacle point cloud data, and then performing principal component analysis and oriented bounding box processing on the second animal-shaped obstacle point cloud data, the distance information between the animal-shaped obstacle and the target intelligent vehicle can be obtained quickly and accurately.

[0149] Specifically, based on the principal orientation obtained through principal component analysis of the second animal-shaped obstacle point cloud data, a first oriented bounding box with the smallest volume that tightly encloses the animal-shaped obstacle (e.g., a person) point cloud can be constructed. The coordinates of the center point of this first oriented bounding box (X... center_p, Y center_p Z center_pIt can accurately reflect the spatial position of animal-shaped obstacles in the target intelligent vehicle's coordinate system, Z. center_p This refers to the distance between the animal-shaped obstacle and the target intelligent vehicle.

[0150] For each point cloud data point of the first non-animal obstacle, the distance information between the corresponding non-animal obstacle and the target intelligent vehicle is obtained through the following steps:

[0151] Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first non-animal obstacle point cloud data is spatially transformed to obtain the second non-animal obstacle point cloud data in the target intelligent vehicle base coordinate system.

[0152] Principal component analysis and oriented bounding box processing were performed sequentially on the point cloud data of the second non-animal type obstacle to obtain the second oriented bounding box;

[0153] Based on the Z-axis coordinates of the center point of the second directional bounding box, the distance information between the corresponding non-animal-shaped obstacle and the target intelligent vehicle is obtained.

[0154] Therefore, by first transforming the point cloud data of the first non-animal obstacle from the color camera coordinate system to the base coordinate system of the target intelligent vehicle to obtain the point cloud data of the second non-animal obstacle, and then performing principal component analysis and oriented bounding box processing on the point cloud data of the second non-animal obstacle, the distance information between the non-animal obstacle and the target intelligent vehicle can be obtained quickly and accurately.

[0155] Specifically, based on the principal orientation obtained through principal component analysis of the second non-animal obstacle point cloud data, a second oriented bounding box with a minimum volume that tightly encloses the non-animal obstacle point cloud can be constructed. The center point coordinates of this second oriented bounding box (X... center_o ,Y center_o Z center_o It can accurately reflect the spatial position of non-animal obstacles in the target intelligent vehicle's coordinate system, Z. center_o This refers to the distance between non-animal-type obstacles and the target intelligent vehicle.

[0156] Specifically, the basic idea of ​​principal component analysis (PCA) is to project the original data into a new coordinate system through linear transformation. This coordinate system is composed of the main directions of the data. For point cloud data, PCA can identify the main directions of the point cloud and thus calculate a minimum bounding box aligned with those main directions.

[0157] For a given point cloud First, calculate the centroid of the point cloud. :

[0158]

[0159] Then, the point cloud is centralized:

[0160]

[0161] Next, calculate the covariance matrix of the centered point cloud:

[0162]

[0163] Then, perform eigenvalue decomposition on the covariance matrix to obtain the eigenvectors of the feature points:

[0164]

[0165] The point cloud is transformed using a new coordinate system composed of feature vectors, and the AABB (axis-aligned bounding box) in the new coordinate system is calculated, which is the OBB (oriented bounding box).

[0166] In some exemplary implementations, based on the distance information between the obstacle and the target intelligent vehicle and a preset graded obstacle avoidance strategy, the target intelligent vehicle is controlled to perform corresponding obstacle avoidance actions, including:

[0167] If the distance between the animal-type obstacle (the animal-type obstacle closest to the target intelligent vehicle) and the target intelligent vehicle, or the distance between the non-animal-type obstacle closest to the target intelligent vehicle and the target intelligent vehicle, is less than or equal to the first preset distance, then the target intelligent vehicle is controlled to perform a preset prohibited area action.

[0168] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle are both greater than the first preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the second preset distance, then control the target intelligent vehicle to perform the preset warning zone action.

[0169] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform the preset safe zone action.

[0170] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a third preset distance, then the target intelligent vehicle is controlled to perform a preset special safe zone action.

[0171] Among them, the first preset distance is less than the second preset distance, and the second preset distance is less than the third preset distance.

[0172] Therefore, by adopting a four-zone (restricted zone, warning zone, safe zone, and extra-safe zone) differentiated response strategy based on the distance between animal-type obstacles (such as people) and non-animal-type obstacles and the target intelligent vehicle, the operational efficiency of the target intelligent vehicle can be guaranteed while ensuring safety.

[0173] It should be noted that, as those skilled in the art will understand, the present invention does not limit the specific values ​​of the first preset distance, the second preset distance, and the third preset distance. The specific values ​​of the first preset distance, the second preset distance, and the third preset distance can be set according to actual obstacle avoidance requirements. For example, the first preset distance can be set to 1.5 meters, the second preset distance can be set to 2.5 meters, and the third preset distance can be set to 3.5 meters.

[0174] In some exemplary implementations, controlling the target intelligent vehicle to perform actions within a preset prohibited area includes:

[0175] Control the target intelligent vehicle to perform emergency stop and alarm actions.

[0176] Therefore, by controlling the target intelligent vehicle to perform emergency stop and alarm actions when the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to a first preset distance (e.g., 1.5 meters), the safety of the staff can be prioritized while ensuring operational efficiency.

[0177] Specifically, an alarm can be triggered via an audible and visual alarm system (e.g., a 96dB buzzer + an orange flashing warning light). Please continue to refer to [the relevant documentation / reference]. Figure 2 This is a schematic diagram of a preset hierarchical obstacle avoidance strategy provided in one embodiment of the present invention. Figure 2As shown, when a non-animal obstacle or an animal obstacle (such as a person) is detected within a first preset distance (e.g., within ≤1.5 meters) from the target intelligent vehicle, it can be determined that the target intelligent vehicle is in a prohibited area. At this time, a three-level emergency mechanism can be triggered: the drive system immediately performs a millisecond-level emergency stop (braking distance ≤0.2m@1m / s, that is, under the condition that the vehicle is traveling at a speed of 1 m / s, the distance the vehicle slides from the triggering of the emergency stop to the complete stop of the vehicle does not exceed 0.2 meters), and activates a 96dB buzzer and an orange strobe warning light (compliant with ISO 13849 PLd level), which can better ensure the safety of the staff while ensuring work efficiency.

[0178] In some exemplary embodiments, if the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a first preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to a second preset distance, then the target intelligent vehicle is controlled to perform a preset warning zone action, including:

[0179] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the second preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0180] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than a first preset distance and less than or equal to a second preset distance, then the target intelligent vehicle is controlled to perform an emergency stop.

[0181] Therefore, this setup can ensure the operational efficiency of the target intelligent vehicle while further protecting the safety of the staff.

[0182] It should be noted that, as those skilled in the art will understand, the present invention does not limit the specific value range of the preset speed range, and the specific value range of the preset speed range can be, but is not limited to, 0.25m / s to 0.35m / s.

[0183] Please continue to refer to this. Figure 2 ,like Figure 2As shown, when a non-animal or animal-shaped obstacle is detected within a range of a first preset distance to a second preset distance from the target intelligent vehicle (e.g., 1.5 meters to 2.5 meters), and no non-animal or animal-shaped obstacle (e.g., a person) is detected within a range of the first preset distance from the target intelligent vehicle (e.g., ≤1.5 meters), it can be determined that the target intelligent vehicle is in a warning zone. Further, if a non-animal-shaped obstacle is detected in the warning zone but no animal-shaped obstacle (e.g., a person) is detected, a dynamic speed suppression strategy is adopted, and the vehicle speed is limited to a preset speed range (e.g., 0.25 m / s to 0.35 m / s) through a PID controller (proportional-integral-derivative controller); if an animal-shaped obstacle (e.g., a person, with a confidence level ≥85%) is detected in the warning zone, an emergency stop response is immediately initiated to ensure the safety of personnel.

[0184] In some exemplary embodiments, if the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to a third preset distance, then the target intelligent vehicle is controlled to perform a preset safe zone action, including:

[0185] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle or the distance between the nearest non-animal obstacle and the target intelligent vehicle is less than or equal to the third preset distance, then the global path of the target intelligent vehicle will be replanned, and the target intelligent vehicle will be controlled to maintain the preset operating speed.

[0186] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, then the target intelligent vehicle will be controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0187] Therefore, this setup can further ensure the safety of staff while maintaining the operational efficiency of the target intelligent vehicle.

[0188] Please continue to refer to this. Figure 2 ,like Figure 2As shown, when a non-animal obstacle or an animal obstacle (e.g., a person) is detected within a range of the second to third preset distances from the target intelligent vehicle (e.g., 2.5 meters to 3.5 meters), and no non-animal or animal obstacles are detected within a range of the second preset distance from the target intelligent vehicle (e.g., ≤2.5 meters), the target intelligent vehicle can be determined to be in a safe area. Further, if a non-animal obstacle is detected within the safe area but no animal obstacle is detected, path replanning is performed while maintaining the operating speed; if an animal obstacle (e.g., a person) is detected within the safe area, a dynamic speed suppression strategy is adopted, and the vehicle speed is limited to a preset speed range (e.g., 0.25 m / s to 0.35 m / s) using a PID controller (proportional-integral-derivative controller).

[0189] In some exemplary implementations, controlling the target intelligent vehicle to perform actions within a preset safe zone includes:

[0190] Control the target intelligent vehicle to maintain the preset operating speed.

[0191] Therefore, by controlling the target intelligent vehicle to maintain a preset operating speed when the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a third preset distance (e.g., greater than 3.5 meters), the safety of the staff can be further guaranteed while ensuring the operating efficiency of the target intelligent vehicle.

[0192] It should be noted that, as those skilled in the art will understand, the present invention does not limit the specific value of the preset operating speed, and the specific value of the preset operating speed can be set according to actual operating requirements.

[0193] Please continue to refer to this. Figure 2 ,like Figure 2 As shown, when no non-animal or animal-shaped obstacles are detected within a third preset distance from the target intelligent vehicle (e.g., within ≤3.5 meters), it can be determined that the target intelligent vehicle is in a special safe zone. At this time, the target intelligent vehicle can maintain the preset operating speed except in special circumstances.

[0194] To further enhance safety, while maintaining the preset operating speed of the target intelligent vehicle, the movement trend of animal-shaped obstacles can be continuously monitored. When an animal-shaped obstacle (such as a person) is detected in the safe area, a dynamic speed suppression strategy is implemented to slowly reduce the operating speed of the target intelligent vehicle until the animal-shaped obstacle (such as a person) moves into the warning area, at which point an emergency stop response is initiated. If the animal-shaped obstacle (such as a person) moves further into the prohibited area, an alarm is also triggered (a 96dB buzzer is activated and an orange strobe warning light flashes).

[0195] Based on the same inventive concept, this invention also provides an intelligent vehicle obstacle avoidance system, please refer to [reference needed]. Figure 3 This is a block diagram of an intelligent vehicle obstacle avoidance system provided in one embodiment of the present invention. Figure 3 As shown, the intelligent vehicle obstacle avoidance system provided by the present invention includes: an image acquisition module 110, configured to acquire a color image and a depth image of the scene where the target intelligent vehicle is located; a conversion module 120, configured to perform spatial conversion on the depth image to obtain first point cloud data in the depth camera coordinate system, and to perform spatial conversion on the first point cloud data according to the spatial conversion relationship between the pre-calibrated depth camera coordinate system and the color camera coordinate system to obtain second point cloud data in the color camera coordinate system; an obstacle detection module 130, configured to detect obstacles in the color image to obtain at least one obstacle region of interest, and to process the second point cloud data according to the obstacle region of interest to obtain corresponding first obstacle point cloud data, and to obtain distance information between the corresponding obstacle and the target intelligent vehicle according to the first obstacle point cloud data; and a graded obstacle avoidance module 140, configured to control the target intelligent vehicle to perform corresponding obstacle avoidance actions according to the distance information between the nearest animal-type obstacle and the target intelligent vehicle, the distance information between the nearest non-animal-type obstacle and the target intelligent vehicle, and a preset graded obstacle avoidance strategy.

[0196] The intelligent vehicle obstacle avoidance system provided by this invention can automatically adopt different obstacle avoidance strategies for obstacles at different distances. This not only effectively reduces human intervention but also significantly improves the safety and intelligence of intelligent vehicles (such as unmanned forklifts) during operation. Furthermore, the intelligent vehicle obstacle avoidance system provided by this invention boasts high real-time performance, achieving a detection efficiency of 15 frames per second (i.e., a detection efficiency of 15fps), ensuring millisecond-level response latency and meeting high real-time obstacle avoidance requirements. In addition, the intelligent vehicle obstacle avoidance system provided by this invention exhibits high detection sensitivity: it can effectively identify small-volume irregular obstacles (minimum cross-sectional area ≥ 5cm²) within 3 meters, stably detect medium-volume irregular obstacles (minimum cross-sectional area ≥ 30cm²) within 5 meters, and reliably perceive large-volume obstacles (minimum cross-sectional area ≥ 1m²) within 7 meters. Simultaneously, the intelligent vehicle obstacle avoidance system provided by this invention possesses strong obstacle avoidance capabilities, achieving an effective obstacle avoidance success rate of over 98%, with a dynamic response time of less than 70 milliseconds.

[0197] In some exemplary embodiments, the graded obstacle avoidance module 140 is also configured to provide a visual parameter adjustment interface for users to adjust a first preset distance, a second preset distance, and a third preset distance.

[0198] Therefore, by providing a visual parameter adjustment interface, users can adjust the first preset distance, the second preset distance, and / or the third preset distance via a host computer or a PAD (tablet) matched with the target intelligent vehicle, thereby realizing a user-configurable intelligent safety protection system.

[0199] In some exemplary embodiments, the transformation module 120 is configured to perform spatial transformation on the depth image using an integer multiplication and displacement operation strategy to obtain first point cloud data in the depth camera coordinate system.

[0200] Please continue to refer to this. Figure 3 ,like Figure 3 As shown, in some exemplary embodiments, the obstacle detection module 130 includes: an animal obstacle detection unit 131 configured to detect animal obstacles in the color image, including humans and other animals besides humans; and a non-animal obstacle detection unit 132 configured to detect non-animal obstacles in the color image.

[0201] In some exemplary embodiments, the animal obstacle detection unit 131 is configured to: convert a pre-trained animal obstacle detection model into RKNN format, perform pruning and quantization operations on the animal obstacle detection model during the conversion process, and run the RKNN format animal obstacle detection model on the NPU to detect animal obstacles in the color image.

[0202] In some exemplary embodiments, the non-animal obstacle detection unit 132 is configured to: convert a pre-trained non-animal obstacle detection model into RKNN format, perform pruning and quantization operations on the non-animal obstacle detection model during the conversion process, and run the RKNN format non-animal obstacle detection model on the NPU to detect non-animal obstacles in the color image.

[0203] In some exemplary implementations, both the animal-type obstacle detection model and the non-animal-type obstacle detection model are YOLOv8s-seg models.

[0204] In some exemplary implementations, the obstacle conversion module is configured to: crop the second point cloud data according to the region of interest of the obstacle to obtain the initial obstacle point cloud data; and perform downsampling, outlier removal and Euclidean clustering operations on the initial obstacle point cloud data in sequence to obtain the first obstacle point cloud data.

[0205] In some exemplary embodiments, the obstacle detection module 130 is configured to: perform spatial transformation on the first obstacle point cloud data according to the pre-calibrated spatial transformation relationship between the color camera coordinate system and the target intelligent vehicle base coordinate system to obtain the second obstacle point cloud data in the target intelligent vehicle base coordinate system; perform principal component analysis and oriented bounding box processing on the second obstacle point cloud data in sequence to obtain the corresponding oriented bounding box; and obtain the distance information between the corresponding obstacle and the target intelligent vehicle according to the Z-axis coordinate value of the center point of the oriented bounding box.

[0206] In some exemplary implementations, the graded obstacle avoidance module 140 is configured as follows:

[0207] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the target intelligent vehicle and the nearest non-animal-shaped obstacle is less than or equal to the first preset distance, then control the target intelligent vehicle to perform a preset prohibited area action.

[0208] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle are both greater than the first preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the second preset distance, then control the target intelligent vehicle to perform the preset warning zone action.

[0209] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform the preset safe zone action.

[0210] If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than a third preset distance, then the target intelligent vehicle is controlled to perform a preset special safe zone action.

[0211] Among them, the first preset distance is less than the second preset distance, and the second preset distance is less than the third preset distance.

[0212] In some exemplary implementations, controlling the target intelligent vehicle to perform actions within a preset prohibited area includes:

[0213] Control the target intelligent vehicle to perform emergency stop and alarm actions.

[0214] In some exemplary implementations, the graded obstacle avoidance module 140 is configured as follows:

[0215] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the second preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0216] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than a first preset distance and less than or equal to a second preset distance, then the target intelligent vehicle is controlled to perform an emergency stop.

[0217] In some exemplary implementations, the graded obstacle avoidance module 140 is configured as follows:

[0218] If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the third preset distance, then the global path of the target intelligent vehicle will be replanned, and the target intelligent vehicle will be controlled to maintain the preset operating speed.

[0219] If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, then the target intelligent vehicle will be controlled to adopt a dynamic speed suppression strategy to reduce the operating speed to within the preset speed range.

[0220] In some exemplary implementations, controlling the target intelligent vehicle to perform actions within a preset safe zone includes:

[0221] Control the target intelligent vehicle to maintain the preset operating speed.

[0222] Based on the same inventive concept, the present invention also provides an electronic device, please refer to... Figure 4 This is a block diagram of an electronic device provided in one embodiment of the present invention. Figure 4 As shown, the electronic device includes a processor 210 and a memory 230. The memory 230 stores a computer program. When the computer program is executed by the processor 210, it implements the intelligent vehicle obstacle avoidance method described above. Since the electronic device provided by this invention and the intelligent vehicle obstacle avoidance method provided by this invention belong to the same inventive concept, the electronic device provided by this invention has at least all the beneficial effects of the intelligent vehicle obstacle avoidance method provided by this invention. Therefore, the beneficial effects of the electronic device provided by this invention can be referred to the relevant descriptions of the beneficial effects of the intelligent vehicle obstacle avoidance method provided by this invention above, and will not be repeated here.

[0223] Please continue to refer to this. Figure 4 ,like Figure 4 As shown, the electronic device also includes a communication interface 220 and a communication bus 240. The processor 210, communication interface 220, and memory 230 communicate with each other via the communication bus 240. The communication bus 240 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 240 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 220 is used for communication between the aforementioned electronic device and other devices.

[0224] It should be noted that the processor 210 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 210 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines.

[0225] It should also be noted that the memory 230 can be used to store computer programs. The processor 210 implements various functions of the electronic device by running or executing the computer programs stored in the memory 230 and by accessing data stored in the memory 230. The memory 230 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable memory (PROM), electrically programmable memory (EPROM), electrically erasable programmable memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, random access memory is available in a variety of forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous random access memory (SDRAM), dual data rate synchronous random access memory (DDRSDRAM), enhanced synchronous random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), memory bus direct random access memory (RDRAM), direct memory bus dynamic random access memory (DRDRAM), and memory bus dynamic random access memory (RDRAM), etc.

[0226] Furthermore, the electronic device can be a mobile embedded device, which can be equipped with an RK3588 chip as the main processor, with built-in 6TOPs computing power, support for INT8 / INT16 quantization, run a custom YOLOv8-seg model (inference latency of 8ms after quantization), use 32GB LPDDR4X memory, and use 64GB eMMC 5.1 + 128GB NVMe SSD for storage (for data caching).

[0227] In summary, compared with the prior art, the intelligent vehicle obstacle avoidance method, system, and electronic device provided by the present invention have the following beneficial effects:

[0228] The intelligent vehicle obstacle avoidance method provided by this invention first acquires color images (e.g., RGB images) and depth images of the scene where the target intelligent vehicle (e.g., an unmanned forklift) is located. Then, obstacle point cloud data is obtained based on the color and depth images. Next, distance information between the obstacle and the target intelligent vehicle is obtained based on the obstacle point cloud data. Finally, graded obstacle avoidance is performed based on the distance information between the obstacle and the target intelligent vehicle. This allows for the automatic adoption of different obstacle avoidance strategies for obstacles at different distances, effectively reducing human intervention and improving the safety and intelligence of intelligent vehicles (e.g., unmanned forklifts) during operation. Furthermore, the intelligent vehicle obstacle avoidance method provided by this invention has high real-time performance, achieving a detection efficiency of 15 frames per second (i.e., a detection efficiency of 15fps), ensuring millisecond-level response latency and meeting high real-time obstacle avoidance requirements. In addition, the intelligent vehicle obstacle avoidance method provided by this invention has high detection sensitivity: it can effectively identify small-volume irregular obstacles (minimum cross-sectional area ≥ 5cm²) within 3 meters, stably detect medium-volume irregular obstacles (minimum cross-sectional area ≥ 30cm²) within 5 meters, and reliably perceive large-volume obstacles (minimum cross-sectional area ≥ 1m²) within 7 meters. Meanwhile, the intelligent vehicle obstacle avoidance method provided by this invention has strong obstacle avoidance capabilities, achieving an effective obstacle avoidance success rate of over 98%, and a dynamic response time of less than 70 milliseconds.

[0229] Furthermore, by running the RKNN format animal obstacle detection model and non-animal obstacle detection model on the NPU (Neural Processing Unit), the present invention can reduce the occupation of CPU (Central Processing Unit) resources, thereby meeting real-time requirements and effectively improving the detection accuracy of animal and non-animal obstacles.

[0230] Furthermore, by transforming floating-point operations into integer multiplication and bitwise operations, this invention can not only effectively improve conversion efficiency and save CPU resources (reducing CPU usage from 85% to 45%), thus solving the problem of excessive resource consumption, but also has little impact on the frame rate of image output, thereby providing a stable and efficient data source for subsequent input data.

[0231] It should be noted that computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0232] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0233] It should also be noted that the above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. An intelligent vehicle obstacle avoidance method, characterized in that, include: Acquire color and depth images of the scene where the target intelligent vehicle is located; The depth image is spatially transformed to obtain first point cloud data in the depth camera coordinate system. Based on the pre-calibrated spatial transformation relationship between the depth camera coordinate system and the color camera coordinate system, the first point cloud data is spatially transformed to obtain second point cloud data in the color camera coordinate system. Obstacle detection is performed on the color image to obtain at least one region of interest for an obstacle; The second point cloud data is processed according to the region of interest of the obstacle to obtain the corresponding first obstacle point cloud data; Based on the first obstacle point cloud data, obtain the distance information between the corresponding obstacle and the target intelligent vehicle; Based on the distance information between the obstacle and the target intelligent vehicle and the preset hierarchical obstacle avoidance strategy, the target intelligent vehicle is controlled to perform corresponding obstacle avoidance actions.

2. The intelligent vehicle obstacle avoidance method according to claim 1, characterized in that, The obstacle detection of the color image includes: The color image is then used to detect animal-type obstacles and non-animal-type obstacles, whereby animal-type obstacles include humans and other animals besides humans.

3. The intelligent vehicle obstacle avoidance method according to claim 2, characterized in that, The step of controlling the target intelligent vehicle to perform corresponding obstacle avoidance actions based on the distance information between the obstacle and the target intelligent vehicle and a preset graded obstacle avoidance strategy includes: If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the target intelligent vehicle and the nearest non-animal-shaped obstacle is less than or equal to a first preset distance, then the target intelligent vehicle is controlled to perform a preset prohibited area action. If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle and the target intelligent vehicle, and the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle, are both greater than the first preset distance, and the distance between the nearest animal-shaped obstacle and the target intelligent vehicle or the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle is less than or equal to the second preset distance, then the target intelligent vehicle is controlled to perform a preset warning zone action; If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle and the target intelligent vehicle, and the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle, are both greater than the second preset distance, and the distance between the nearest animal-shaped obstacle and the target intelligent vehicle or the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform a preset safe zone action; If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the third preset distance, then the target intelligent vehicle is controlled to perform a preset special safety zone action. Wherein, the first preset distance is less than the second preset distance, and the second preset distance is less than the third preset distance.

4. The intelligent vehicle obstacle avoidance method according to claim 3, characterized in that, The control of the target intelligent vehicle to perform actions in a preset prohibited area includes: Control the target intelligent vehicle to perform emergency stop and alarm actions.

5. The intelligent vehicle obstacle avoidance method according to claim 3, characterized in that, If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle, are both greater than the first preset distance, and the distance between the nearest animal-shaped obstacle and the target intelligent vehicle, or the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle, are less than or equal to the second preset distance, then the target intelligent vehicle is controlled to perform a preset warning zone action, including: If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the second preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce its operating speed to a preset speed range. If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, then the target intelligent vehicle is controlled to perform an emergency stop.

6. The intelligent vehicle obstacle avoidance method according to claim 3, characterized in that, If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, and the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the second preset distance, and the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the nearest non-animal-shaped obstacle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform a preset safety zone action, including: If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the third preset distance, then the global path of the target intelligent vehicle is replanned, and the target intelligent vehicle is controlled to maintain a preset operating speed. If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce its operating speed to a preset speed range.

7. The intelligent vehicle obstacle avoidance method according to claim 3, characterized in that, The control of the target intelligent vehicle to perform actions within a preset safe zone includes: Control the target intelligent vehicle to maintain a preset operating speed.

8. The intelligent vehicle obstacle avoidance method according to claim 2, characterized in that, The detection of animal-type obstacles and non-animal-type obstacles in the color image includes: The pre-trained animal obstacle detection model and the non-animal obstacle detection model are both converted into RKNN format, and pruning and quantization operations are performed on the animal obstacle detection model and the non-animal obstacle detection model during the conversion process. The animal-type obstacle detection model and the non-animal-type obstacle detection model in RKNN format are run on the NPU to detect animal-type obstacles and non-animal-type obstacles in the color image, respectively.

9. The intelligent vehicle obstacle avoidance method according to claim 8, characterized in that, Both the animal-type obstacle detection model and the non-animal-type obstacle detection model are YOLOv8s-seg models.

10. The intelligent vehicle obstacle avoidance method according to claim 1, characterized in that, The step of processing the second point cloud data according to the region of interest of the obstacle to obtain the corresponding first obstacle point cloud data includes: The second point cloud data is cropped according to the region of interest of the obstacle to obtain the initial obstacle point cloud data; The initial obstacle point cloud data is subjected to downsampling, outlier removal, and Euclidean clustering operations in sequence to obtain the first obstacle point cloud data.

11. The intelligent vehicle obstacle avoidance method according to claim 1, characterized in that, The step of obtaining the distance information between the corresponding obstacle and the target intelligent vehicle based on the first obstacle point cloud data includes: Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first obstacle point cloud data is spatially transformed to obtain the second obstacle point cloud data in the target intelligent vehicle base coordinate system. Principal component analysis and oriented bounding box processing are performed sequentially on the second obstacle point cloud data to obtain the corresponding oriented bounding boxes; Based on the Z-axis coordinate value of the center point of the directional bounding box, the distance information between the corresponding obstacle and the target intelligent vehicle is obtained.

12. The intelligent vehicle obstacle avoidance method according to claim 1, characterized in that, The step of spatially transforming the depth image to obtain the first point cloud data in the depth camera coordinate system includes: The depth image is spatially transformed using integer multiplication and displacement operations to obtain the first point cloud data in the depth camera coordinate system.

13. An intelligent vehicle obstacle avoidance system, characterized in that, include: The image acquisition module is configured to acquire color and depth images of the scene where the target intelligent vehicle is located; The conversion module is configured to perform spatial conversion on the depth image to obtain first point cloud data in the depth camera coordinate system, and to perform spatial conversion on the first point cloud data according to the pre-calibrated spatial conversion relationship between the depth camera coordinate system and the color camera coordinate system to obtain second point cloud data in the color camera coordinate system. An obstacle detection module is configured to detect obstacles in the color image to obtain at least one obstacle region of interest, process the second point cloud data according to the obstacle region of interest to obtain corresponding first obstacle point cloud data, and obtain distance information between the corresponding obstacle and the target intelligent vehicle according to the first obstacle point cloud data. as well as The graded obstacle avoidance module is configured to control the target intelligent vehicle to perform corresponding obstacle avoidance actions based on the distance information between the obstacle and the target intelligent vehicle and a preset graded obstacle avoidance strategy.

14. The intelligent vehicle obstacle avoidance system according to claim 13, characterized in that, The obstacle detection module includes: An animal obstacle detection unit is configured to detect animal obstacles in the color image, wherein the animal obstacles include humans and other animals besides humans; and A non-animal obstacle detection unit is configured to detect non-animal obstacles in the color image.

15. The intelligent vehicle obstacle avoidance system according to claim 14, characterized in that, The hierarchical obstacle avoidance module is configured as follows: If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle or the target intelligent vehicle and the nearest non-animal-shaped obstacle is less than or equal to a first preset distance, then the target intelligent vehicle is controlled to perform a preset prohibited area action. If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle and the target intelligent vehicle, and the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle, are both greater than the first preset distance, and the distance between the nearest animal-shaped obstacle and the target intelligent vehicle or the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle is less than or equal to the second preset distance, then the target intelligent vehicle is controlled to perform a preset warning zone action; If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle and the target intelligent vehicle, and the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle, are both greater than the second preset distance, and the distance between the nearest animal-shaped obstacle and the target intelligent vehicle or the distance between the nearest non-animal-shaped obstacle and the target intelligent vehicle is less than or equal to the third preset distance, then the target intelligent vehicle is controlled to perform a preset safe zone action; If the distance between the target intelligent vehicle and the nearest animal-shaped obstacle and the target intelligent vehicle, as well as the distance between the target intelligent vehicle and the nearest non-animal-shaped obstacle, are both greater than the third preset distance, then the target intelligent vehicle is controlled to perform a preset special safety zone action. Wherein, the first preset distance is less than the second preset distance, and the second preset distance is less than the third preset distance.

16. The intelligent vehicle obstacle avoidance system according to claim 15, characterized in that, The control of the target intelligent vehicle to perform actions in a preset prohibited area includes: Control the target intelligent vehicle to perform emergency stop and alarm actions.

17. The intelligent vehicle obstacle avoidance system according to claim 15, characterized in that, The hierarchical obstacle avoidance module is configured as follows: If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the second preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce its operating speed to a preset speed range. If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the first preset distance and less than or equal to the second preset distance, then the target intelligent vehicle is controlled to perform an emergency stop.

18. The intelligent vehicle obstacle avoidance system according to claim 15, characterized in that, The hierarchical obstacle avoidance module is configured as follows: If the distance between the nearest non-animal obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, and the distance between the nearest animal obstacle and the target intelligent vehicle is greater than the third preset distance, then the global path of the target intelligent vehicle is replanned, and the target intelligent vehicle is controlled to maintain a preset operating speed. If the distance between the nearest animal-shaped obstacle and the target intelligent vehicle is greater than the second preset distance and less than or equal to the third preset distance, then the target intelligent vehicle is controlled to adopt a dynamic speed suppression strategy to reduce its operating speed to a preset speed range.

19. The intelligent vehicle obstacle avoidance system according to claim 15, characterized in that, The control of the target intelligent vehicle to perform actions within a preset safe zone includes: Control the target intelligent vehicle to maintain a preset operating speed.

20. The intelligent vehicle obstacle avoidance system according to claim 14, characterized in that, The animal obstacle detection unit is configured to: convert a pre-trained animal obstacle detection model into RKNN format, perform pruning and quantization operations on the animal obstacle detection model during the conversion process, and run the animal obstacle detection model in RKNN format on the NPU to detect animal obstacles in the color image. The non-animal obstacle detection unit is configured to: convert a pre-trained non-animal obstacle detection model into RKNN format, perform pruning and quantization operations on the non-animal obstacle detection model during the conversion process, and run the RKNN format non-animal obstacle detection model on the NPU to detect non-animal obstacles in the color image.

21. The intelligent vehicle obstacle avoidance system according to claim 20, characterized in that, Both the animal-type obstacle detection model and the non-animal-type obstacle detection model are YOLOv8s-seg models.

22. The intelligent vehicle obstacle avoidance system according to claim 13, characterized in that, The obstacle conversion module is configured as follows: The second point cloud data is cropped according to the region of interest of the obstacle to obtain the initial obstacle point cloud data; The initial obstacle point cloud data is subjected to downsampling, outlier removal, and Euclidean clustering operations in sequence to obtain the first obstacle point cloud data.

23. The intelligent vehicle obstacle avoidance system according to claim 13, characterized in that, The obstacle detection module is configured as follows: Based on the spatial transformation relationship between the pre-calibrated color camera coordinate system and the target intelligent vehicle base coordinate system, the first obstacle point cloud data is spatially transformed to obtain the second obstacle point cloud data in the target intelligent vehicle base coordinate system. Principal component analysis and oriented bounding box processing are performed sequentially on the second obstacle point cloud data to obtain the corresponding oriented bounding boxes; Based on the Z-axis coordinate value of the center point of the directional bounding box, the distance information between the corresponding obstacle and the target intelligent vehicle is obtained.

24. The intelligent vehicle obstacle avoidance system according to claim 13, characterized in that, The conversion module is configured to perform spatial conversion on the depth image using integer multiplication and displacement operation strategies to obtain the first point cloud data in the depth camera coordinate system.

25. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the intelligent vehicle obstacle avoidance method as described in any one of claims 1 to 12.

Images