Method and device for detecting travelable area in orchard scene

By using visual sensors and deep learning technology in orchard scenes to combine typical obstacles with atypical obstacles, the accuracy of driving area detection in orchard scenes is solved, and the safety and operation of unmanned driving systems are improved efficiency.

CN119027900APending Publication Date: 2024-11-26LOVOL HEAVY IND CO LTD

Patent Information

Application Number
CN202411002142.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In complex orchard scenes, existing unmanned driving technology is difficult to accurately detect possible areas, especially the ability to identify unknown obstacles and small targets is limited, which poses safety risks.

Method used

By installing vision sensors in front of the vehicle, we collect road environment images in real time, and use computer vision and deep learning technology for analysis, combining the fusion method of typical obstacles and atypical obstacles, we can detect the driving areas in the orchard scene to improve the detection Accuracy and reliability.

Benefits of technology

It greatly improves the perception ability of the orchard in complex environments, improves the performance of driving area detection in unmanned driving systems, provides more reliable and accurate detection results, and enhances the operation efficiency and safety guarantee of autonomous vehicles in orchard scenarios. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027900A_ABST
    Figure CN119027900A_ABST
Patent Text Reader

Abstract

The invention provides a method and a device for detecting a drivable area in an orchard scene. The method comprises the following steps: acquiring an initial road environment image of a road in front of a vehicle collected by at least one predetermined visual sensor in the orchard scene; for each initial road environment image, preprocessing the initial road environment image to obtain a target road environment image; inputting the target road environment image into a drivable area detection model, and determining an initial drivable area and a typical obstacle area in the target road environment image; and fusing the initial drivable area in the target road environment image with the typical obstacle area to obtain a target drivable area in the target road environment image. Through the method and the device, the accuracy of detection of the drivable area in the orchard scene is improved, the working efficiency of the automatic driving vehicle in the orchard scene is further improved, and meanwhile, a reliable safety guarantee is provided for unmanned driving of orchard agricultural machinery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and more particularly to a method and device for detecting a drivable area in an orchard scenario. Background Art

[0002] With the continuous development of science and technology, the unmanned driving technology of agricultural machinery is gradually moving towards intelligence and industrialization. More and more people are paying attention to how to realize precise unmanned driving technology in complex orchard scene environments to assist in weeding, spraying and other operations in orchards and reduce labor.

[0003] At present, most unmanned driving technologies are mainly used in structured environments such as urban roads, where the road geometry features are obvious, such as lane lines and traffic signs, which are conducive to the environmental perception and decision-making control of unmanned driving systems. However, in unstructured environments such as orchards, the roads are irregular, the background is complex, and there are many types of obstacles, which bring great challenges to unmanned driving systems.

[0004] At present, the commonly used safety strategy methods for unmanned agricultural machinery mainly include two types: vision-based typical obstacle recognition and lidar-based atypical obstacle information recognition. The vision-based obstacle recognition method can only target known obstacles in the whitelist. It is difficult to effectively detect unknown obstacles or areas, which poses a safety hazard. Although the lidar-based obstacle recognition method can solve the problem of drivable area detection to a certain extent, it is limited by hardware performance and has limited perception capabilities for small target spaces. Therefore, how to improve the accuracy of drivable area detection in orchard scenarios has become a technical issue that cannot be underestimated. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a detection method and device for the drivable area in an orchard scene, by installing a visual sensor in front of the vehicle, collecting road environment images in real time, using computer vision and deep learning technology to analyze the images, and detecting the drivable area of ​​the road in front of the vehicle in the orchard scene in real time, and by fusing typical obstacles with atypical obstacles, a more reliable and accurate drivable area in the orchard is obtained. The accuracy of drivable area detection in the orchard scene is improved, the operating efficiency of autonomous driving vehicles in the orchard scene is further improved, and reliable safety protection is provided for unmanned driving of orchard agricultural machinery.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting a drivable area in an orchard scene, the detection method comprising:

[0007] Acquire an initial road environment image of a road ahead of the vehicle collected by at least one predetermined visual sensor in an orchard scene;

[0008] For each initial road environment image, preprocessing the initial road environment image to obtain a target road environment image;

[0009] The target road environment image is input into a pre-trained drivable area detection model to determine an initial drivable area and a typical obstacle area in the target road environment image;

[0010] The initial drivable area in the target road environment image is fused with the typical obstacle area to obtain a target drivable area in the target road environment image.

[0011] Furthermore, after obtaining the target drivable area in the target road environment image, the detection method further includes:

[0012] Divide the target road environment image into regions according to a preset partitioning rule, and determine a plurality of road regions in the target road environment image;

[0013] For each road area, the proportion of the non-feasible area in the road area is determined based on the target drivable area in the target road environment image, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

[0014] Furthermore, the controlling the driving state of the vehicle based on the proportion of the non-feasible area in the road area includes:

[0015] When the road area is a first preset area, and the proportion of the non-feasible area in the road area is greater than or equal to a first proportion threshold, the vehicle is controlled to perform emergency braking until the vehicle stops; wherein the first preset area represents a road section within a first distance range in front of the vehicle;

[0016] When the road area is a second preset area and the proportion of the non-feasible area in the road area is greater than or equal to a second proportion threshold, the vehicle is controlled to decelerate, and an obstacle avoidance trajectory is planned based on the non-feasible area in the road area so that the vehicle travels along the obstacle avoidance trajectory; wherein the second preset area represents a road section within a second distance range in front of the vehicle.

[0017] Furthermore, the preprocessing of the initial road environment image to obtain the target road environment image includes:

[0018] Offline calibration is performed on the initial road environment image to obtain distortion correction parameters, and based on the distortion correction parameters, the initial road environment image is corrected to obtain a road environment corrected image;

[0019] The road environment correction image is smoothed, and the attribute parameters of the road environment correction image are adjusted based on the display effect of the road environment correction image to obtain the target road environment image.

[0020] Furthermore, the initial drivable area in the target road environment image is merged with the typical obstacle area to obtain a target drivable area in the target road environment image, including:

[0021] Comparing the segmentation mask of the initial drivable area with the segmentation mask of the typical obstacle area to determine a repeated area;

[0022] Eliminating the repeated area from the initial drivable area to obtain a fused drivable area;

[0023] The segmentation mask of the fused drivable area is post-processed to obtain the target drivable area in the target road environment image; wherein the post-processing includes removing isolated area processing and smoothing area boundary processing.

[0024] Furthermore, the drivable area detection model adopts receptive field attention convolution operation and lightweight convolution operation.

[0025] In a second aspect, an embodiment of the present application further provides a device for detecting a drivable area in an orchard scene, the detection device comprising:

[0026] An image acquisition module, used to acquire an initial road environment image of a road ahead of a vehicle collected by at least one predetermined visual sensor in an orchard scene;

[0027] An image processing module is used to pre-process each initial road environment image to obtain a target road environment image;

[0028] An area detection module is used to input the target road environment image into a pre-trained drivable area detection model to determine an initial drivable area and a typical obstacle area in the target road environment image;

[0029] The area fusion module is used to fuse the initial drivable area in the target road environment image with the typical obstacle area to obtain a target drivable area in the target road environment image.

[0030] Furthermore, the detection device further includes a decision module. After obtaining the target drivable area in the target road environment image, the decision module is used to:

[0031] Divide the target road environment image into regions according to a preset partitioning rule, and determine a plurality of road regions in the target road environment image;

[0032] For each road area, the proportion of the non-feasible area in the road area is determined based on the target drivable area in the target road environment image, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

[0033] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for detecting a drivable area in an orchard scenario as described above are performed.

[0034] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for detecting a drivable area in an orchard scenario as described above are executed.

[0035] A method and device for detecting a drivable area in an orchard scene provided in an embodiment of the present application include: first, obtaining an initial road environment image of a road ahead of a vehicle captured by at least one predetermined visual sensor in an orchard scene; then, for each initial road environment image, preprocessing the initial road environment image to obtain a target road environment image; inputting the target road environment image into a pre-trained drivable area detection model to determine an initial drivable area and a typical obstacle area in the target road environment image; finally, fusing the initial drivable area and the typical obstacle area in the target road environment image to obtain the target drivable area in the target road environment image.

[0036] This application installs a visual sensor in front of the vehicle to collect road environment images in real time, uses computer vision and deep learning technology to analyze the images, and detects the drivable area of ​​the road in front of the vehicle in the orchard scene in real time, which greatly improves the perception ability in the complex environment of the orchard and improves the performance of drivable area detection in the orchard unmanned driving system. And through the method of fusing typical obstacles with atypical obstacles, the drivable area detected by the drivable area detection model is fused with the typical obstacle area to further obtain a more reliable and accurate drivable area of ​​the orchard. Compared with other networks that only detect typical obstacles or only detect drivable areas, the detection results obtained by the detection method provided by this application are more accurate and more robust. The accuracy of drivable area detection in the orchard scene is improved, the operating efficiency of the autonomous driving vehicle in the orchard scene is further improved, and at the same time, reliable safety protection is provided for unmanned driving of orchard agricultural machinery.

[0037] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a method for detecting a drivable area in an orchard scenario provided in an embodiment of the present application;

[0040] Figure 2 One of the structural schematic diagrams of a detection device for a drivable area in an orchard scene provided in an embodiment of the present application;

[0041] Figure 3 A second structural schematic diagram of a device for detecting a drivable area in an orchard scenario provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0044] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of autonomous driving technology.

[0045] With the continuous development of science and technology, the unmanned driving technology of agricultural machinery is gradually moving towards intelligence and industrialization. More and more people are paying attention to how to realize precise unmanned driving technology in complex orchard scene environments to assist in weeding, spraying and other operations in orchards and reduce labor.

[0046] Research has found that at present, most unmanned driving technologies are mainly used in structured environments such as urban roads, where the road geometry features are obvious, such as lane lines and traffic signs, which are conducive to environmental perception and decision-making control of unmanned driving systems. However, in unstructured environments such as orchards, the roads are irregular, the background is complex, and there are many types of obstacles, which pose great challenges to unmanned driving systems. For example, orchard roads may be scattered with agricultural tools and equipment, and there may be ditches, railings and other uncertain factors that affect driving, which are difficult for unmanned driving systems to accurately identify. In order to improve the safety and reliability of unmanned driving of agricultural machinery in orchard environments, it is crucial to accurately detect the drivable area on orchard roads.

[0047] At present, the commonly used safety strategy methods for unmanned agricultural machinery mainly include typical obstacle recognition based on vision and atypical obstacle information recognition based on lidar. The image in front of the agricultural machinery is captured by a camera, and then image processing and pattern recognition algorithms are used to detect and identify obstacles. This method has a good recognition effect for typical obstacles with obvious visual features such as color, shape and texture, such as people and agricultural machinery, but it cannot implement safety strategies for unclear feature information and uncertain factors in the scene. The obstacle recognition method based on lidar measures the distance of the object by emitting lasers and receiving reflected signals, thereby constructing a three-dimensional point cloud map of the surrounding environment. By segmenting and clustering the point cloud data, irregular obstacles can be identified. Lidar has strong robustness under complex lighting conditions, but due to the complex orchard scene and the large number of noise points, the recognition effect of the drivable area of ​​the road ahead needs to be improved. If the information recognition of the forward process is not clear, there will be a safety strategy of mistaken parking.

[0048] The vision-based obstacle recognition method can only identify known obstacles in the whitelist. It is difficult to effectively detect unknown obstacles or areas (such as ditches, orchard branches, stones, railings, etc.), which poses a safety hazard. Although the obstacle recognition method based on lidar can solve the problem of drivable area detection to a certain extent, it is limited by hardware performance and has limited perception capabilities for small target spaces. In addition, lidar is expensive and has difficulties in commercial promotion in the field of agricultural machinery. In addition, the existing methods lack special designs for complex orchard scenes, making it difficult to fully utilize prior knowledge of the orchard environment, resulting in insufficient recognition accuracy and reliability.

[0049] In summary, the existing safety strategies for unmanned driving of agricultural machinery still face many challenges when dealing with the complex environment of orchards. Therefore, how to improve the accuracy of drivable area detection in orchard scenarios has become a technical issue that cannot be underestimated.

[0050] Based on this, an embodiment of the present application provides a method for detecting a drivable area in an orchard scenario, which improves the accuracy of detecting the drivable area in the orchard scenario, further improves the operating efficiency of autonomous driving vehicles in the orchard scenario, and at the same time provides reliable safety guarantees for unmanned driving of orchard agricultural machinery.

[0051] See also Figure 1 , Figure 1 This is a flow chart of a method for detecting a drivable area in an orchard scene provided by an embodiment of the present application. Figure 1 As shown in, the detection method provided in the embodiment of the present application includes:

[0052] S101, acquiring an initial road environment image of a road ahead of a vehicle captured by at least one predetermined visual sensor in an orchard scene.

[0053] Here, the predetermined visual sensor can be installed at the front end of the unmanned vehicle, and a color camera (such as CCD, CMOS, etc.) can be used, which is not specifically limited in this application. When installing the predetermined visual sensor, the vehicle coordinate system is used as a reference and installed at a fixed position and a fixed angle. The number and layout of the predetermined visual sensors can be optimized according to factors such as the complexity of the orchard scene environment and the size of the blind area of ​​the field of view, so as to expand the perception range as much as possible and reduce the omission of obstacles. The predetermined visual sensor is used to collect a high-resolution RGB image sequence of the road environment in front of the vehicle in real time.

[0054] For the above step S101, in the specific implementation, an initial road environment image of the road ahead of the vehicle collected by at least one predetermined visual sensor in the orchard scene is obtained. Here, the road environment image captured by the predetermined visual sensor has rich texture and color features; considering the large changes in the illumination of the orchard scene, the predetermined visual sensors are equipped with functions such as automatic exposure and automatic white balance, and use wide dynamic technology to adapt to the imaging requirements of different weather and different time periods. The collected initial road environment image is published by the ROS node.

[0055] S102 : For each initial road environment image, preprocess the initial road environment image to obtain a target road environment image.

[0056] Regarding the above step S102, in a specific implementation, after a plurality of initial road environment images are acquired in step S101, each initial road environment image is preprocessed to obtain a target road environment image.

[0057] As an optional embodiment, with respect to the above step S102, the preprocessing of the initial road environment image to obtain the target road environment image includes:

[0058] Step 1021 , off-line calibration is performed on the initial road environment image to obtain distortion correction parameters, and the initial road environment image is corrected based on the distortion correction parameters to obtain a road environment corrected image.

[0059] Here, image offline calibration refers to the process of calibrating the parameters of the image acquisition device in image processing. The purpose of calibration is to eliminate image distortion and determine the conversion relationship between the image coordinate system and the world coordinate system, thereby improving the accuracy and reliability of image processing.

[0060] For the above step 1021, in the specific implementation, the initial road environment image is first calibrated offline, for example, the Zhang Zhengyou calibration method can be used to calibrate the internal and external parameters of the camera offline to obtain distortion correction parameters. Then, the distortion correction parameters are applied to the initial road environment image collected in real time, and the initial road environment image is corrected based on the distortion correction parameters to obtain a road environment correction image to eliminate the error introduced by the lens distortion.

[0061] Step 1022: Smoothing the road environment correction image, and adjusting the attribute parameters of the road environment correction image based on the display effect of the road environment correction image to obtain the target road environment image.

[0062] In order to suppress the noise mixed in the imaging process, the image needs to be smoothed. For the above step 1022, in the specific implementation, the road environment correction image obtained in step 1021 is smoothed. For example, the road environment correction image can be smoothed by using methods such as median filtering and bilateral filtering, while retaining important features such as edges, removing the interference of salt and pepper noise and Gaussian noise. Then, based on the display effect of the road environment correction image, the attribute parameters of the road environment correction image are adjusted according to the conditions to obtain the target road environment image. Here, the attribute parameters can be parameters such as contrast, brightness, saturation, etc. of the road environment correction image, which are not specifically limited in this application. For example, by means of histogram equalization, Gamma correction, etc., the distinction between the drivable area and the non-drivable area in the road environment correction image is highlighted. The above-mentioned operations of preprocessing the image are all implemented using highly optimized CUDA programming to meet real-time requirements.

[0063] In this way, according to the above steps 1021-1022, the quality of perception data is enhanced through image preprocessing, and a series of preprocessing is performed on the collected initial road environment image to eliminate errors and noise in the image, highlight the distinction between the drivable area and the non-drivable area in the road environment correction image, prepare for the subsequent drivable area detection, and improve the accuracy of the subsequent drivable area detection.

[0064] S103, inputting the target road environment image into a pre-trained drivable area detection model to determine an initial drivable area and a typical obstacle area in the target road environment image.

[0065] Here, the drivable area detection model uses an instance segmentation algorithm based on deep learning. Due to the diversity of materials, colors, and textures in the road area, this application uses a powerful and fast end-to-end convolutional neural network as an instance segmentation model to maximize the use of road appearance and contextual prior knowledge. Considering the complexity of the orchard scene, the drivable area detection model can also perform instance segmentation detection on some typical obstacles in the target road environment image to obtain typical obstacle areas. Typical obstacles include: people, agricultural machinery, ditches and other obstacles.

[0066] For the above step S103, in the specific implementation, after obtaining the target road environment image, the target road environment image is input into a pre-trained drivable area detection model to determine the initial drivable area and typical obstacle area in the target road environment image, and the orchard drivable area and obstacle area can be obtained from the target road environment image.

[0067] Since the perception of drivable areas and typical obstacles requires a larger receptive field to make judgments on the global background in order to make more accurate inference results, here, according to the embodiments provided in the present application, the drivable area detection model uses a receptive field attention convolution operation (RFAConv) and a lightweight convolution operation Slim-neck. The receptive field attention mechanism enhances the utilization of global information, and the lightweight convolution reduces the complexity of the model. Specifically, the target algorithm in the drivable area detection model is improved through receptive field attention convolution operations and lightweight convolution operations. As an example, when the target algorithm in the drivable area detection model is yolov8-seg, the above two algorithms can be added to yolov8-seg for model improvement; when the target algorithm in the drivable area detection model is fasterrcnn, the above two algorithms are added to fasterrcnn for model improvement to achieve better detection effects.

[0068] The receptive field attention convolution operation not only emphasizes the importance of different features within the receptive field slider, but also gives priority to the receptive field spatial features. In this way, the problem of convolution kernel parameter sharing is completely solved. RFA not only pays attention to the receptive field spatial features, but also provides effective attention weights for large-size convolution kernels. The computational cost and parameter increment provided by the receptive field attention convolution operation are almost negligible, while significantly improving the network performance. In general, the calculation of RFA can be expressed as the following formula:

[0069] F = Softmax(g 1×1 (AvgPool()))×ReLU(Norm(g k×k ()))= rf ×F rf

[0070] Among them, g 1×1 represents the size of 1×1, k represents the size of the convolution kernel, Norm represents normalization, X represents the input target road environment image, and F is obtained by converting the attention map A rf And the transformed receptive field spatial feature F rf Obtained by multiplication.

[0071] In order to further improve the running efficiency of the algorithm on the vehicle edge computing platform, a lightweight convolution operation Slim-neck is used to reduce the weight of the model while maintaining accuracy. Slim-neck is a structure designed to optimize the neck part of the convolutional neural network. It consists of GSConv, which then GSConv forms GSBottleneck and VoVGSCSP modules, which solves the problem that lightweight models built with a large number of deeply separable convolutional layers cannot achieve sufficient accuracy.

[0072] Here, according to the embodiment provided by the present application, the trained drivable area detection model file is deployed on the domain controller, and GPU accelerated reasoning is implemented using the CUDA and TensorRT frameworks. Each frame of the target road environment image input is first cropped and scaled to a size that matches the network input, and then fed into the segmentation network to predict the category pixel by pixel, and then the resulting data is changed to the original resolution to obtain the initial drivable area in the target road environment image.

[0073] As an optional embodiment, the drivable area detection model is trained through the following steps:

[0074] (1): Obtain multiple road environment sample images in orchard scenes.

[0075] Among them, each road environment sample image has a corresponding drivable sample area and obstacle sample area.

[0076] In the specific implementation of step (1), a plurality of road environment sample images with drivable sample areas and obstacle sample areas are obtained. Here, the road environment sample images can be road environment images collected by a predetermined visual sensor in a historical time.

[0077] (2): Input a plurality of the road environment sample images into the original detection model of the drivable area to determine the drivable prediction area and obstacle prediction area corresponding to each road environment sample image.

[0078] Regarding the above step (2), during the specific implementation, multiple road environment sample images are input into the drivable area original detection model. The drivable area original detection model performs drivable area detection and obstacle area detection on each road environment sample image, and determines the drivable prediction area and obstacle prediction area corresponding to each road environment sample image.

[0079] (3): Compare the drivable sample area corresponding to each road environment sample image with the drivable predicted area, compare the obstacle sample area corresponding to each road environment sample image with the obstacle predicted area, and calculate the loss value of the original detection model of the drivable area in the current state.

[0080] (4): Based on the loss value of the original detection model of the drivable area, the model parameters of the original detection model of the drivable area are continuously adjusted until the original detection model of the drivable area reaches a convergence state, thereby obtaining the image classification model.

[0081] Here, the loss value is an indicator of the learning model during the training process, which is used to evaluate the error between the model prediction results and the true value.

[0082] Regarding the above steps (3) to (4), during the specific implementation, for each road environment sample image, the drivable sample area of ​​the road environment sample image is compared with the drivable prediction area, and the obstacle sample area is compared with the obstacle prediction area. By comparing the drivable sample area of ​​the road environment sample image with the drivable prediction area, and comparing the obstacle sample area of ​​the road environment sample image with the obstacle prediction area, it is determined whether the detection of the original detection model of the drivable area is accurate. When the drivable sample area of ​​the road environment sample image is different from the drivable prediction area, or the obstacle sample area is different from the obstacle prediction area, it is considered that the detection of the original detection model of the drivable area is inaccurate. At this time, it is necessary to calculate the loss value of the original detection model of the drivable area in the current state. The method of calculating the loss value is described in detail in the prior art and will not be described in detail here. Then, the model parameters of the original detection model of the drivable area are continuously adjusted based on the loss value. The original detection model of the drivable area will continuously minimize the loss through iteration. At each step of the iteration, the loss value of the original detection model of the drivable area will be calculated. When the loss value of the original detection model of the drivable area cannot reach the loss threshold, the model parameters of the original detection model of the drivable area will be continuously updated. The new parameters will calculate the new loss value, so that the loss value shows a fluctuating downward trend during the iteration process. Finally, when the loss value reaches smoothness, that is, when the loss value of the trained original detection model of the drivable area is not greater than the loss threshold, that is, when the loss value is not significantly reduced compared with the loss value calculated last time, it is considered that the original detection model of the drivable area has reached a convergence state. At this time, the detection of the original detection model of the drivable area is relatively accurate. At this time, the training is terminated and the drivable area detection model is obtained.

[0083] In this way, according to the above step S103, the deep learning technology is combined with the computer vision technology to improve the performance of the drivable area detection in the orchard unmanned driving system. Through massive data training, the deep learning model can learn rich scene prior knowledge and understand the contextual information in the orchard environment, so as to make more accurate detection when facing complex and changeable actual scenes. By introducing deep learning algorithms for drivable area detection, drivable area detection is achieved in a purely visual way, getting rid of the dependence on high-cost lidar, overcoming the limitations of traditional visual methods, and greatly improving the perception ability in the complex environment of the orchard. The instance segmentation network can fully tap the scene prior knowledge and accurately identify roads and non-road areas. Based on mature and inexpensive industrial cameras and embedded computing platforms, after engineering optimization, it provides a reliable basis for unmanned driving decision-making, which can fully meet the requirements of agricultural machinery for cost performance.

[0084] S104: Fusing the initial drivable area in the target road environment image with the typical obstacle area to obtain a target drivable area in the target road environment image.

[0085] For the above step S104, in the specific implementation, after obtaining the initial drivable area and the typical obstacle area in the target road environment image, the initial drivable area in the target road environment image is fused with the typical obstacle area to obtain the target drivable area in the target road environment image. In this way, the detection method provided in the embodiment of the present application fuses the detected initial drivable area with the typical obstacle area to obtain the final target drivable area. Compared with other networks that only detect typical obstacles or only detect drivable areas, the detection results obtained by the detection method provided by the present application are more accurate and more robust.

[0086] As an optional embodiment, with respect to the above step S104, the merging of the initial drivable area in the target road environment image with the typical obstacle area to obtain the target drivable area in the target road environment image includes:

[0087] Step 1041 : Compare the segmentation mask of the initial drivable area with the segmentation mask of the typical obstacle area to determine the overlapping area.

[0088] Step 1042: remove the repeated area from the initial drivable area to obtain a fused drivable area.

[0089] In the specific implementation of the above steps 1041 to 1042, the segmentation mask of the initial drivable area in the target road environment image is compared with the segmentation mask of the typical obstacle area to determine the repeated area. Then, the repeated area is removed from the initial drivable area in the target road environment image to obtain a fused drivable area.

[0090] Step 1043 , post-processing the fused segmentation mask of the drivable area to obtain the target drivable area in the target road environment image.

[0091] Among them, post-processing includes removing isolated areas and smoothing area boundaries.

[0092] Regarding the above step 1043, in the specific implementation, the segmentation mask of the fused drivable area is post-processed, where the post-processing includes removing isolated area processing and smoothing area boundary processing to obtain the target drivable area in the target road environment image.

[0093] In this way, according to the above steps 1041 to 1043, the identified initial drivable area is fused with the typical obstacle area data. Specifically, the segmentation mask of the obstacle area is compared with the segmentation mask of the drivable area. For the overlapping area, it is removed from the drivable area to obtain a more accurate and safe representation of the drivable area. The proposal of the drivable area can prevent the vehicle from hitting some atypical obstacles during automatic driving, such as rare atypical objects such as animals and different agricultural tools. Combined with the perception of typical obstacles, the common obstacles have a further grasp. The two perception results are fused to further obtain a more reliable and accurate drivable area of ​​the orchard. The fused drivable area mask is then post-processed, such as removing small isolated areas and smoothing area boundaries, to improve the robustness and reliability of the results. The probability of typical obstacles being misclassified into the drivable area is further reduced, making the unmanned driving system safer and more reliable.

[0094] As an optional embodiment, after obtaining the target drivable area in the target road environment image, the detection method provided by the present application further includes:

[0095] A: The target road environment image is divided into regions according to a preset partitioning rule, and a plurality of road regions are determined in the target road environment image.

[0096] In the specific implementation of step A, the target road environment image is divided into regions according to a preset partitioning rule, and multiple road regions are determined in the target road environment image. Here, as an example, the vehicle forward direction region in the target road environment image can be divided into three regions: near, middle, and far, representing the road sections of 0-3m, 3-7m, and 7-10m directly in front of the vehicle, respectively.

[0097] Here, it should be noted that the above-mentioned method of dividing the target road environment image into regions is only an example. In fact, the method of dividing the target road environment image into regions is not limited to the above-mentioned example.

[0098] B: For each road area, the proportion of the non-feasible area in the road area is determined based on the target drivable area in the target road environment image, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

[0099] For the above step B, during the specific implementation, after dividing multiple road areas, for each road area, the drivable area contained in the road area is determined based on the target drivable area in the target road environment image to determine the proportion of the non-feasible area in the road area, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

[0100] As an optional embodiment, with respect to the above step B, controlling the driving state of the vehicle based on the proportion of the non-feasible area in the road area includes:

[0101] a: When the road area is a first preset area and the proportion of the non-feasible area in the road area is greater than or equal to a first proportion threshold, the vehicle is controlled to perform emergency braking until the vehicle stops.

[0102] Here, the first preset area represents a road section located within a first distance range in front of the vehicle. For example, the first distance range can be set to 0-3m, which is not specifically limited in this application.

[0103] For the above step a, in the specific implementation, when the road area is the first preset area and the proportion of the non-feasible area in the road area is greater than or equal to the first proportion threshold, it means that there is an obstacle at a position close to the vehicle and the area is impassable. At this time, the vehicle is controlled to perform emergency braking until the vehicle stops.

[0104] b: When the road area is a second preset area and the proportion of the non-feasible area in the road area is greater than or equal to a second proportion threshold, the vehicle is controlled to decelerate, and an obstacle avoidance trajectory is planned based on the non-feasible area in the road area so that the vehicle travels along the obstacle avoidance trajectory.

[0105] Here, the second preset area represents a road section located within a second distance range in front of the vehicle. For example, the second distance range can be set to 3-7m, which is not specifically limited in this application.

[0106] For the above step b, in the specific implementation, when the road area is the second preset area and the proportion of the non-feasible area in the road area is greater than or equal to the second proportion threshold, it means that there is still a certain distance between the obstacle and the vehicle. At this time, the vehicle is controlled to slow down, and the obstacle avoidance trajectory is planned based on the non-feasible area in the road area, so that the vehicle can travel along the obstacle avoidance trajectory.

[0107] In this way, according to the above steps a-b, based on the semantic segmentation results, taking into account the relative distance between the obstacle and the vehicle, the traffic safety of the road ahead of the vehicle is evaluated in real time, and decisions such as start-stop, steering, and obstacle avoidance are made. When there is an obstacle at a position close to the vehicle, the vehicle is immediately controlled to perform emergency braking. When the obstacle is still at a certain distance from the vehicle, the vehicle is controlled to avoid the obstacle and interact with the unmanned driving system to achieve an emergency obstacle avoidance function, thereby improving the safety of the vehicle's automatic driving. As an optional embodiment, in order to smooth the vehicle's motion response and avoid frequent starts and stops, the detection method provided in this application can also track obstacles in the time domain. When an obstacle appears in the segmentation mask for multiple consecutive frames (such as 10 frames), the vehicle obstacle avoidance decision is triggered to filter out false detections caused by isolated noise or short-term occlusion, while ensuring safety, the vehicle's motion continuity and stability are improved.

[0108] The method for detecting a drivable area in an orchard scene provided in an embodiment of the present application first obtains an initial road environment image of a road ahead of a vehicle captured by at least one predetermined visual sensor in an orchard scene; then, for each initial road environment image, the initial road environment image is preprocessed to obtain a target road environment image; the target road environment image is input into a pre-trained drivable area detection model to determine an initial drivable area and a typical obstacle area in the target road environment image; finally, the initial drivable area in the target road environment image is fused with the typical obstacle area to obtain the target drivable area in the target road environment image.

[0109] This application installs a visual sensor in front of the vehicle to collect road environment images in real time, uses computer vision and deep learning technology to analyze the images, and detects the drivable area of ​​the road in front of the vehicle in the orchard scene in real time, which greatly improves the perception ability in the complex environment of the orchard and improves the performance of drivable area detection in the orchard unmanned driving system. And through the method of fusing typical obstacles with atypical obstacles, the drivable area detected by the drivable area detection model is fused with the typical obstacle area to further obtain a more reliable and accurate drivable area of ​​the orchard. Compared with other networks that only detect typical obstacles or only detect drivable areas, the detection results obtained by the detection method provided by this application are more accurate and more robust. The accuracy of drivable area detection in the orchard scene is improved, the operating efficiency of the autonomous driving vehicle in the orchard scene is further improved, and at the same time, reliable safety protection is provided for unmanned driving of orchard agricultural machinery.

[0110] See also Figure 2 and Figure 3 , Figure 2 One of the structural schematic diagrams of a detection device for a drivable area in an orchard scene provided in an embodiment of the present application; Figure 3 This is a second structural diagram of a detection device for a drivable area in an orchard scene provided in an embodiment of the present application. Figure 2 As shown in , the detection device 200 includes:

[0111] An image acquisition module 201 is used to acquire an initial road environment image of a road ahead of a vehicle collected by at least one predetermined visual sensor in an orchard scene;

[0112] The image processing module 202 is used to pre-process each initial road environment image to obtain a target road environment image;

[0113] The area detection module 203 is used to input the target road environment image into a pre-trained drivable area detection model to determine the initial drivable area and typical obstacle area in the target road environment image;

[0114] The area fusion module 204 is used to fuse the initial drivable area in the target road environment image with the typical obstacle area to obtain a target drivable area in the target road environment image.

[0115] Further, such as Figure 3 As shown, the detection device 200 further includes a decision module 205. After obtaining the target drivable area in the target road environment image, the decision module 205 is used to:

[0116] Divide the target road environment image into regions according to a preset partitioning rule, and determine a plurality of road regions in the target road environment image;

[0117] For each road area, the proportion of the non-feasible area in the road area is determined based on the target drivable area in the target road environment image, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

[0118] Furthermore, when the decision module 205 is used to control the driving state of the vehicle based on the proportion of the non-feasible area in the road area, the decision module 205 is also used to:

[0119] When the road area is a first preset area, and the proportion of the non-feasible area in the road area is greater than or equal to a first proportion threshold, the vehicle is controlled to perform emergency braking until the vehicle stops; wherein the first preset area represents a road section within a first distance range in front of the vehicle;

[0120] When the road area is a second preset area and the proportion of the non-feasible area in the road area is greater than or equal to a second proportion threshold, the vehicle is controlled to decelerate, and an obstacle avoidance trajectory is planned based on the non-feasible area in the road area so that the vehicle travels along the obstacle avoidance trajectory; wherein the second preset area represents a road section within a second distance range in front of the vehicle.

[0121] Furthermore, when the image processing module 202 is used to pre-process the initial road environment image to obtain the target road environment image, the image processing module 202 is also used to:

[0122] Offline calibration is performed on the initial road environment image to obtain distortion correction parameters, and based on the distortion correction parameters, the initial road environment image is corrected to obtain a road environment corrected image;

[0123] The road environment correction image is smoothed, and the attribute parameters of the road environment correction image are adjusted based on the display effect of the road environment correction image to obtain the target road environment image.

[0124] Further, when the region fusion module 204 is used to fuse the initial drivable area in the target road environment image with the typical obstacle area to obtain the target drivable area in the target road environment image, the region fusion module 204 is also used to:

[0125] Comparing the segmentation mask of the initial drivable area with the segmentation mask of the typical obstacle area to determine a repeated area;

[0126] Eliminating the repeated area from the initial drivable area to obtain a fused drivable area;

[0127] The segmentation mask of the fused drivable area is post-processed to obtain the target drivable area in the target road environment image; wherein the post-processing includes removing isolated area processing and smoothing area boundary processing.

[0128] Furthermore, the drivable area detection model adopts receptive field attention convolution operation and lightweight convolution operation.

[0129] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0130] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the method for detecting the drivable area in the orchard scene in the method embodiment shown in the embodiment, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0131] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the method for detecting the drivable area in the orchard scene in the method embodiment shown in the embodiment, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0136] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0137] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for detecting a drivable area in an orchard scene, characterized in that: The detection method comprises: Acquiring an initial road environment image of a road ahead of the vehicle captured by at least one predetermined visual sensor in an orchard scene; For each initial road environment image, preprocess the initial road environment image to obtain a target road environment image; Inputting the target road environment image into a pre-trained drivable area detection model to determine the initial drivable area and typical obstacle areas in the target road environment image; The initial drivable area in the target road environment image is fused with the typical obstacle area to obtain a target drivable area in the target road environment image.

2. The detection method according to claim 1, wherein After obtaining the target drivable area in the target road environment image, the detection method further includes: Dividing the target road environment image into regions according to a preset partitioning rule, and determining a plurality of road regions in the target road environment image; For each road area, the proportion of the non-feasible area in the road area is determined based on the target drivable area in the target road environment image, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

3. The detection method according to claim 2, characterized in that The controlling of the driving state of the vehicle based on the proportion of the non-feasible area in the road area includes: When the road area is a first preset area and the proportion of the non-feasible area in the road area is greater than or equal to a first proportion threshold, controlling the vehicle to perform emergency braking until the vehicle stops; wherein the first preset area is a road section within a first distance range in front of the vehicle; When the road area is a second preset area and the proportion of the non-feasible area in the road area is greater than or equal to a second proportion threshold, the vehicle is controlled to decelerate, and an obstacle avoidance trajectory is planned based on the non-feasible area in the road area so that the vehicle travels along the obstacle avoidance trajectory; wherein the second preset area represents a road section located within a second distance range in front of the vehicle.

4. The detection method according to claim 1, wherein The preprocessing of the initial road environment image to obtain the target road environment image includes: Performing offline calibration on the initial road environment image to obtain distortion correction parameters, and correcting the initial road environment image based on the distortion correction parameters to obtain a road environment corrected image; The road environment correction image is smoothed, and attribute parameters of the road environment correction image are adjusted based on a display effect of the road environment correction image to obtain the target road environment image.

5. The detection method according to claim 1, wherein The fusing the initial drivable area in the target road environment image with the typical obstacle area to obtain a target drivable area in the target road environment image includes: Comparing the segmentation mask of the initial drivable area with the segmentation mask of the typical obstacle area to determine the overlapping area; Eliminating the repeated area from the initial drivable area to obtain a fused drivable area; The segmentation mask of the fused drivable area is post-processed to obtain the target drivable area in the target road environment image; wherein the post-processing includes removing isolated areas and smoothing area boundaries.

6. The detection method according to claim 1, characterized in that The drivable area detection model adopts receptive field attention convolution operation and lightweight convolution operation.

7. A device for detecting a drivable area in an orchard scene, characterized in that: The detection device comprises: An image acquisition module, configured to acquire an initial road environment image of a road ahead of the vehicle captured by at least one predetermined visual sensor in an orchard scene; An image processing module is used to pre-process each initial road environment image to obtain a target road environment image; An area detection module is used to input the target road environment image into a pre-trained drivable area detection model to determine the initial drivable area and typical obstacle areas in the target road environment image; The region fusion module is used to fuse the initial drivable region in the target road environment image with the typical obstacle region to obtain a target drivable region in the target road environment image.

8. The detection device according to claim 7, characterized in that The detection device further includes a decision module, which is configured to: Dividing the target road environment image into regions according to a preset partitioning rule, and determining a plurality of road regions in the target road environment image; For each road area, the proportion of the non-feasible area in the road area is determined based on the target drivable area in the target road environment image, and the driving state of the vehicle is controlled based on the proportion of the non-feasible area in the road area.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus. When the processor is running, the machine-readable instructions execute the steps of the method for detecting a drivable area in an orchard scenario as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting a drivable area in an orchard scene according to any one of claims 1 to 6 are executed.

Citation Information

Patent Citations

  • Drivable region detection method, drivable region detection device, and electronic equipment

    CN106485233A

  • Trinocular rearview mirror and trinocular vision safe driving method and system

    CN110321877A

  • Parking control method and device, computer equipment and storage medium

    CN114940162A

  • Automobile obstacle avoidance system and method, automatic driving system, vehicle and storage medium

    CN116061968A

  • Vehicle travelable area detection method and device, electronic equipment and storage medium

    CN118038409A

Cited By

  • Obstacle avoidance method and device, self-moving equipment and computer program product

    CN120469414A