Automatic driving method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an automatic driving method and device, electronic equipment, a computer readable storage medium and a computer program product. The automatic driving method comprises the following steps: acquiring Three-Dimensional (3D) point cloud data, and projecting the 3D point cloud data to an image plane to obtain projected point cloud data; the method comprises the following steps: acquiring a first two-dimensional (Two Dimensional, 2D) image; wherein each pixel point in the first 2D image is marked as a drivable area or a non-drivable area; judging whether the projected point cloud data is a drivable area or not based on the first 2D image; and if the point cloud data is the drivable area, the target drivable area is constructed based on the projected point cloud data, so that automatic driving of the vehicle is performed based on the target drivable area, and the accuracy and robustness of drivable area identification are improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving method, device, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] The rapid development of autonomous driving technology has placed higher demands on vehicles' perception and decision-making capabilities in complex environments. Among these, accurate identification of drivable areas is a key aspect of achieving safe navigation. This task typically relies on semantic segmentation of image or point cloud data to determine which areas are suitable for vehicle passage.

[0003] One typical approach in related technologies is to combine 2D images and 3D LiDAR data for autonomous driving control. However, this technology suffers from two main problems: first, the computational load of 3D point cloud data processing is high, limiting real-time performance; second, the data fusion efficiency between 2D images and 3D point clouds is low, affecting the overall system's response speed and stability. Therefore, improving processing efficiency while ensuring accuracy has become a pressing technical challenge. Summary of the Invention

[0004] This application provides an autonomous driving method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy and robustness of drivable area identification while reducing hardware costs and computational complexity.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides an autonomous driving method, including: Acquire 3D point cloud data and project the 3D point cloud data onto an image plane to obtain projected point cloud data; Acquire a first 2D image; wherein each pixel in the first 2D image is marked as a drivable area or a non-drivable area; Based on the first 2D image, it is determined whether the projected point cloud data is a drivable area; if it is a drivable area, a target drivable area is constructed based on the projected point cloud data, so as to perform autonomous driving of the vehicle based on the target drivable area.

[0006] This application provides an autonomous driving device, the device comprising: The acquisition module is used to acquire three-dimensional 3D point cloud data and project the 3D point cloud data onto an image plane to obtain projected point cloud data; and to acquire a first 2D image; wherein each pixel in the first 2D image is marked as a drivable area or a non-drivable area; The judgment module is used to determine whether the projected point cloud data is a drivable area based on the first 2D image; If the construction module is a drivable area, it constructs a target drivable area based on the projected point cloud data, and performs autonomous driving based on the target drivable area.

[0007] In the above scheme, the acquisition module is specifically used to transform the 3D point cloud data through a first transformation matrix to obtain 3D points in the camera coordinate system; and to project the 3D points in the camera coordinate system through a second transformation matrix to obtain the projected point cloud data.

[0008] In the above scheme, the judgment module is specifically used to determine the polygonal region in 3D space based on the horizontal and vertical field of view of the camera mounted on the vehicle; project the polygonal region onto the image plane to obtain the projected polygonal region; and, if the first 2D image is located within the projected polygonal region, determine whether the projected point cloud data is a drivable area based on the first 2D image.

[0009] In the above scheme, the judgment module is further used to perform preset processing on the first 2D image to obtain a processed first 2D image; wherein, the preset processing includes median filtering processing and morphological closing operation processing; to determine whether the projected point cloud data is a drivable area in the processed first 2D image; if the projected point cloud data is a drivable area, then the projected point cloud data is marked as a drivable point.

[0010] In the above scheme, the construction module is specifically used to acquire a second 2D image; wherein the second 2D image includes the first 2D image; query the drivable neighboring pixels that are closest to the non-drivable point in the second 2D image; and construct the target drivable area based on the neighboring pixels and the drivable point.

[0011] In the above scheme, the construction module is further used to mark the 3D point cloud data corresponding to the neighboring pixels as boundary points; and to construct the target drivable area based on the boundary points and the 3D point cloud data corresponding to the drivable points.

[0012] This application provides an electronic device, including: Memory is used to store executable instructions or computer programs. When a processor executes computer-executable instructions or computer programs stored in the memory, it implements an autonomous driving method provided in the embodiments of this application.

[0013] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements an autonomous driving method provided in this application.

[0014] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement an autonomous driving method provided in this application.

[0015] The embodiments of this application have the following beneficial effects: First, 3D point cloud data is projected onto an image plane to obtain projected point cloud data. Then, combined with semantically segmented 2D image information (i.e., the first 2D image), drivability is assessed using the point cloud data, thus achieving fusion processing of 2D and 3D information. This method effectively compensates for the lack of depth information in 2D images and the lack of semantic understanding in 3D point clouds, improving the accuracy and robustness of drivable area recognition while reducing hardware costs and computational complexity. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the autonomous driving method provided in the embodiments of this application. Figure 1 ; Figure 2 This is a flowchart illustrating the autonomous driving method provided in the embodiments of this application. Figure 2 ; Figure 3 This is a schematic diagram of the structure of the autonomous driving device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0019] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0020] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0023] 1) 3D point cloud data: refers to the set of discrete points on the surface of an object in three-dimensional space collected by sensors such as LiDAR. Each point usually contains x, y, and z coordinate information, which is used to describe the spatial position and shape of the object in the environment.

[0024] 2) Image Plane: This is the pixel arrangement structure on a two-dimensional plane corresponding to the camera imaging process. Points collected from three-dimensional space can be projected onto this plane for corresponding analysis with the 2D image.

[0025] 3) Semantic Segmentation: This is a computer vision technique that classifies each pixel in an input image to identify target objects such as drivable areas, non-drivable areas, road boundaries, and obstacles.

[0026] 4) Drivable Area: This refers to the area where a vehicle can safely traverse while driving. It typically includes roads, the area between lane lines, and other ground areas that vehicles are permitted to pass through. Identifying drivable areas is a crucial foundation for autonomous driving systems to perform path planning and obstacle avoidance control.

[0027] 5) Field of View (FOV): This refers to the range that a camera can capture in the horizontal and vertical directions. The horizontal field of view determines the width of the camera's field of view in the left-right direction, while the vertical field of view determines the range of the field of view in the up-down direction.

[0028] The rapid development of autonomous driving technology has placed higher demands on vehicles' perception and decision-making capabilities in complex environments. Among these, accurate identification of drivable areas is a key aspect of achieving safe navigation. This task typically relies on semantic segmentation of image or point cloud data to determine which areas are suitable for vehicle passage.

[0029] One typical approach in related technologies is to combine 2D images and 3D LiDAR data for autonomous driving control. However, this technology suffers from two main problems: first, the computational load of 3D point cloud data processing is high, limiting real-time performance; second, the data fusion efficiency between 2D images and 3D point clouds is low, affecting the overall system's response speed and stability. Therefore, improving processing efficiency while ensuring accuracy has become a pressing technical challenge.

[0030] In view of this, embodiments of this application provide an autonomous driving method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively compensate for the lack of depth information in 2D images and the lack of semantic understanding in 3D point clouds, improve the accuracy and robustness of drivable area recognition, and reduce hardware costs and computational complexity. The autonomous driving method provided by embodiments of this application will be described in detail below.

[0031] See Figure 1 , Figure 1 A flowchart illustrating an autonomous driving method provided in this application embodiment. Figure 1 Specifically, it includes: Step 101: Acquire 3D point cloud data and project the 3D point cloud data onto the image plane to obtain the projected point cloud data.

[0032] It should be noted that, in this embodiment of the application, 3D point cloud data can refer to a set of discrete spatial points in the environment surrounding the target vehicle collected by a lidar sensor. Each point contains three coordinates: x, y, and z, which are used to describe the spatial position and shape of objects in the environment represented by the 3D point cloud data. For example, during the movement of the target vehicle, the lidar sensor emits a laser beam and receives the reflected signal, thereby constructing a high-precision three-dimensional point cloud model that reflects information such as terrain, obstacles, and road boundaries around the target vehicle.

[0033] Optionally, in the embodiments of this application, when projecting 3D point cloud data onto an image plane to obtain projected point cloud data, the 3D point cloud data can be transformed using a first transformation matrix to obtain 3D points in the camera coordinate system; then, the 3D points in the camera coordinate system can be projected using a second transformation matrix to obtain projected point cloud data.

[0034] It should be noted that, in the embodiments of this application, the first transformation matrix may refer to the coordinate transformation matrix used to transform 3D points in the lidar coordinate system to the camera coordinate system, i.e., the lidar-camera extrinsic parameter matrix; the first transformation matrix contains the spatial position and attitude information of the lidar relative to the camera, so that the original point cloud data collected by the lidar can be mapped to the camera's viewpoint, thereby achieving consistency of spatial coordinates.

[0035] It should be noted that, in the embodiments of this application, the point cloud data is transformed using the first transformation matrix, which can unify the data collected by the LiDAR into the same coordinate system as the camera, facilitating subsequent registration and fusion processing with 2D images. For example, in an autonomous driving system, the LiDAR and camera are usually installed in different locations on the vehicle, so precise calibration is required to obtain the first transformation matrix to ensure spatial consistency between the two.

[0036] It should be noted that, in the embodiments of this application, the second transformation matrix can refer to the camera intrinsic parameter matrix. The second transformation matrix describes the camera's internal parameters, including focal length, principal point coordinates, and possible distortion coefficients. Through the second transformation matrix, 3D points in the camera coordinate system can be projected onto the 2D image plane, thereby obtaining the position information of the point cloud in the image.

[0037] In the embodiments of this application, by using a second transformation matrix for projection processing, points in 3D space can be converted into pixel coordinates on an image, thereby achieving the purpose of visualization or further processing on the image. For example, in autonomous driving scenarios, the operation of projection processing using the second transformation matrix can be used to align the LiDAR point cloud with the image segmentation results, so that the corresponding drivable area information in the image can be extracted.

[0038] For example, in the embodiments of this application, the 3D point cloud data can be transformed by the following formula (1) to obtain 3D points in the camera coordinate system. Then, the 3D points in the camera coordinate system can be projected by the camera intrinsic parameter matrix to obtain the projected point cloud data.

[0039] (1) in, Represents a 3D point in the camera coordinate system. Denotes the first transformation matrix. This represents a point in the radar coordinate system.

[0040] In the embodiments of this application, semantic segmentation and region recognition can be performed on the image by mapping high-dimensional 3D point cloud data onto a 2D image plane. This enhances the fusion effect of multimodal data and further enables more accurate drivable area detection and boundary recognition.

[0041] In other words, in this embodiment of the application, by using a first transformation matrix and a second transformation matrix to perform coordinate transformation and projection processing on the 3D point cloud data step by step, the lidar point cloud and the 2D image can be effectively registered and fused. This method can improve the accuracy of drivable area identification and reduce the risk of misjudgment caused by using a single data source, thereby achieving safer and more reliable autonomous driving decision support and further enhancing overall environmental perception capabilities and operational efficiency.

[0042] Step 102: Obtain a first 2D image; wherein each pixel in the first 2D image is marked as a drivable area or a non-drivable area.

[0043] It should be noted that, in the embodiments of this application, the first 2D image may include a binarized image generated after semantic segmentation processing, where each pixel of the binarized image represents information about whether the road is drivable or not. For example, by using a Convolutional Neural Network (CNN) to perform semantic segmentation processing on an image captured by a forward-looking camera, a mask image is output. In the mask image, a pixel value of 1 indicates that the corresponding location belongs to a drivable area, and a pixel value of 0 indicates that the corresponding location belongs to a non-drivable area. The mask image can help understand key information such as lane lines, road boundaries, and obstacles on the road, and provide support for subsequent path planning.

[0044] It should be noted that, in the embodiments of this application, the first 2D image after semantic segmentation (each pixel of the first 2D image represents a binary result of whether it is drivable or not) is the key basis for determining whether the target vehicle is drivable. After each pixel in the first 2D image is processed by a deep learning algorithm, only two states are retained: drivable or not drivable, which greatly simplifies the decision-making logic and improves real-time processing efficiency. For example, in the first 2D image, non-drivable areas such as lane lines, sidewalks, and guardrails are marked as not drivable, while the area inside the lane is marked as drivable, which helps to quickly make path selection.

[0045] Optionally, in embodiments of this application, Figure 2 A flowchart illustrating an autonomous driving method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, after acquiring the first 2D image, i.e., after step 102, the following steps may also be included: Step 104: Determine the polygonal region in 3D space based on the horizontal and vertical field of view of the camera mounted on the vehicle.

[0046] It should be noted that, in the embodiments of this application, the horizontal field of view (HFOV) refers to the angular range that the camera can capture images in the horizontal direction, determining the lateral width of the camera's field of view corresponding to the horizontal field of view; the vertical field of view (VFOV) refers to the camera's angular range in the vertical direction, affecting the camera's ability to cover the scene in the vertical direction corresponding to the vertical field of view. HFOV and VFOV define the camera's visible range in three-dimensional space, which is typically represented as a pyramidal or approximately polyhedral spatial region.

[0047] In this embodiment, the boundary of the visible area can be calculated in 3D space based on the camera's HFOV and VFOV, forming a polygonal region. The polygonal region represents the range of environmental information that the camera can effectively capture in three-dimensional space, ensuring that subsequent processing is only performed on the data within the polygonal region, thereby improving processing efficiency and accuracy.

[0048] In other words, in the embodiments of this application, by using the horizontal and vertical field of view of the camera to calculate the visible area in 3D space, the processing range of the LiDAR point cloud can be more accurately defined, which helps to reduce the interference of invalid data on the processing process and further improve the recognition accuracy and efficiency of the drivable area.

[0049] Step 105: Project the polygonal region onto the image plane to obtain the projected polygonal region.

[0050] It should be noted that, in the embodiments of this application, the image plane refers to the two-dimensional pixel coordinate system formed by camera imaging, which is used to represent the position information of each point in the image. The polygonal region in 3D space is projected and transformed through the camera intrinsic parameter matrix (including parameters such as focal length and principal point offset), and the polygonal region in 3D space is mapped onto the image plane to form a two-dimensional polygonal region. In this way, the geometric structure in three-dimensional space can be converted into a two-dimensional form that is convenient for image processing.

[0051] Step 106: If the first 2D image is located within the polygonal region after projection, determine whether the point cloud data after projection is a drivable area based on the first 2D image.

[0052] For example, in the embodiments of this application, a pixel in the first 2D image is considered to have valid depth information only when it is located within the projected polygonal region, and can be used to further determine whether the point cloud data corresponding to that pixel belongs to a drivable area. By combining the semantic information of the 2D image with the 3D point cloud data, the drivable path in front of the vehicle can be identified more accurately, especially in complex road environments.

[0053] In the embodiments of this application, the visible area in 3D space is calculated based on the horizontal and vertical field of view of the camera, and the visible area is projected onto the image plane, thereby limiting the processing range of point cloud data in the autonomous driving system. Simultaneously, the semantic information of the 2D image is used to analyze the projected point cloud data to determine whether it belongs to a drivable area. This method, combining camera field of view with 2D image semantic information, effectively improves the accuracy and real-time performance of drivable area identification, thereby better supporting vehicle motion planning and obstacle avoidance decisions, and ultimately enhancing the safety and reliability of autonomous driving.

[0054] Step 103: Determine whether the projected point cloud data is a drivable area based on the first 2D image; if it is a drivable area, construct a target drivable area based on the projected point cloud data, and perform autonomous driving of the vehicle based on the target drivable area.

[0055] Optionally, in embodiments of this application, when determining whether the projected point cloud data is a drivable area based on the first 2D image, the first 2D image can be pre-processed to obtain a processed first 2D image; wherein, the pre-processing includes median filtering and morphological closing operation processing; then it can be determined whether the projected point cloud data is a drivable area in the processed first 2D image; if the projected point cloud data is a drivable area, the projected point cloud data is marked as a drivable point.

[0056] It should be noted that, in the embodiments of this application, median filtering is a non-linear digital image processing technique used to remove noise from images, especially salt-and-pepper noise. Median filtering achieves noise reduction by replacing the value of the current pixel with the median of its neighboring pixels. Compared to mean filtering, median filtering more effectively suppresses noise while preserving edge information, making it suitable for semantic segmentation tasks that require maintaining the clarity of road boundaries.

[0057] It should be noted that, in the embodiments of this application, morphological closing operation is an image processing operation that combines dilation and erosion. It is typically used to fill small holes or broken lines in an image, making the segmentation result more continuous and complete. Morphological closing operation first performs dilation to expand the target region, and then performs erosion to restore the original size, thereby eliminating disconnected regions in the segmented image and improving the accuracy of subsequent point cloud matching.

[0058] In the embodiments of this application, the combined processing of median filtering and morphological closing operation can effectively improve the semantic segmentation quality of 2D images, ensuring that the segmented drivable regions are smoother and more coherent, and avoiding misjudgment problems caused by image noise or irregular boundaries.

[0059] For example, in an embodiment of this application, for each point in the 3D point cloud data projected onto the image plane, the label of the corresponding pixel in the processed first 2D image can be checked. If the corresponding pixel is marked as a drivable area, the projected point cloud data is considered to be located within a drivable area; otherwise, it may belong to a non-drivable area and be marked as a non-drivable point.

[0060] It should be noted that, in the embodiments of this application, median filtering and morphological closing operations are introduced to preprocess the 2D image, improving the accuracy and stability of image segmentation; by matching the projected point cloud data with the drivable area in the processed 2D image, effective fusion of 2D and 3D data is achieved; and by marking the projected point cloud data as drivable points, the practicality of drivable area identification is further enhanced. Thus, the method of this application embodiment can effectively overcome the limitations of single sensor data, thereby enabling more reliable and safer autonomous driving environment perception, and providing high-quality data support for path planning and decision control.

[0061] Optionally, in embodiments of this application, when constructing a target drivable area based on projected point cloud data, a second 2D image can be obtained; wherein the second 2D image includes the first 2D image; the nearest drivable neighboring pixels to non-drivable points are queried in the second 2D image; and then the target drivable area can be constructed based on the neighboring pixels and drivable points.

[0062] For example, in embodiments of this application, the second 2D image may be an image obtained by expanding or enhancing the first 2D image, which serves to provide more comprehensive semantic information and spatial context features. The second 2D image may originate from the fusion processing of multiple frames captured by the same camera, from the synthesis processing of data acquired by different sensors (such as infrared sensors and depth cameras), or from an enhanced image generated through model inference. In autonomous driving scenarios, the second 2D image may be an image formed by fusing the original RGB image with the semantic segmentation result, or it may be a new image generated through semantic enhancement processing methods. This application does not specifically limit the method of obtaining the second 2D image.

[0063] It should be noted that, in the embodiments of this application, by introducing a second 2D image, richer environmental information can be obtained, thereby improving the accuracy and robustness of subsequent drivable area identification. As an extended version of the first 2D image, the second 2D image can retain the information in the first 2D image while increasing its adaptability to complex factors such as edges, textures, and lighting changes.

[0064] It should be noted that, in the embodiments of this application, a neighboring pixel can refer to the pixel closest to a certain non-drivable point, which is used to find the nearest drivable area boundary of the non-drivable point. By finding the nearest drivable pixel (i.e., neighboring pixel) to each non-drivable point, the boundary of the drivable area can be determined, and the coherence and completeness of the area division can be further optimized.

[0065] It should be noted that, in the embodiments of this application, there is a close spatial relationship between non-drivable points and their neighboring pixels. The neighborhood search range can be dynamically determined based on the location of the non-drivable point to ensure that no potential drivable areas are missed. Furthermore, the selection strategy for neighboring pixels can be optimized by incorporating image semantic information, such as prioritizing road-related label pixels, thereby improving the accuracy of region boundary determination.

[0066] Optionally, in embodiments of this application, when constructing a target drivable area based on neighboring pixels and drivable points, the 3D point cloud data corresponding to the neighboring pixels can be marked as boundary points; then the target drivable area can be constructed based on the 3D point cloud data corresponding to the boundary points and drivable points.

[0067] It should be noted that, in the embodiments of this application, when neighboring pixels are projected into 3D space, the 3D point cloud data corresponding to these neighboring pixels are regarded as boundary points. The introduction of boundary points can enhance the perception of road boundaries, especially in complex environments such as changes in lighting or occlusion. This approach helps to improve the robustness of identifying drivable areas. For example, in rainy weather or at night, traditional methods may not be able to accurately determine road edges, while boundary points combined with neighboring pixels can provide additional spatial cues, thereby improving the stability of the system.

[0068] It should be noted that, in the embodiments of this application, the 3D point cloud data corresponding to the drivable points can represent the area where the vehicle can safely pass. By combining boundary points and drivable points, the constructed drivable area has higher spatial integrity and accuracy, which can better support the navigation and obstacle avoidance decisions of the autonomous driving system in complex terrain, thereby improving the overall system safety and efficiency.

[0069] Furthermore, in the embodiments of this application, all drivable points can be used as an internal point set, and boundary points can be used as an external contour for fusion processing to finally generate a three-dimensional drivable area description (i.e., the target drivable area), as shown in the following formula (2). The operation of using all drivable points as an internal point set and boundary points as an external contour for fusion processing not only preserves the main part of the drivable area, but also improves the geometric structure of the drivable area by supplementing the boundary points, thereby enabling the path planning module and the obstacle avoidance control module to obtain more accurate input information.

[0070] (2) in, Indicates a drivable area. Represents the set of all drivable points as an interior point set. Indicates the boundary point.

[0071] In other words, in the embodiments of this application, by marking the data corresponding to neighboring pixels as boundary points and combining them with drivable points to construct the target drivable area, road boundary information can be captured more comprehensively, thereby improving the recognition accuracy of the drivable area and enhancing the adaptability and safety of the autonomous driving system in complex environments. Furthermore, the point set can be filtered using bounding boxes to remove points that highly overlap with obstacles, thus ensuring that the final constructed target drivable area is safer and more reliable.

[0072] This embodiment of the application projects 3D point cloud data onto an image plane to obtain projected point cloud data; then, by combining semantically segmented 2D image information (i.e., the first 2D image), drivability is determined from the point cloud data, thereby achieving the fusion processing of 2D and 3D information. This method effectively compensates for the lack of depth information in 2D images and the lack of semantic understanding in 3D point clouds, improving the accuracy and robustness of drivable area recognition while reducing hardware costs and computational complexity.

[0073] Based on the above embodiments, another embodiment of this application provides an autonomous driving method. This method can realize the fusion processing of 2D drivable segmentation region and 3D radar in autonomous driving. The accuracy and real-time performance of drivable region identification directly affect the vehicle's decision-making and motion planning capabilities in dynamic environments. Specifically, based on the given forward-looking camera image, segmentation result S (i.e., the first 2D image), and LiDAR point cloud (i.e., 3D point cloud data), the drivable region and boundary region point set in front of the vehicle can be extracted, and the results can be published to downstream modules (such as navigation modules) in various forms for use or visualization.

[0074] It should be noted that, in the embodiments of this application, the autonomous driving method may include the following steps: Step 1. Data preprocessing and coordinate transformation: Given a frame of laser point cloud (i.e., point cloud data) and the corresponding camera intrinsic parameters (i.e., the second transformation matrix) and radar-camera extrinsic parameter matrix (i.e., the first transformation matrix), the ROS format point cloud data can first be converted into Point Cloud Library (PCL) format, and then the points in the laser coordinate system are projected to the camera coordinate system through the above formula (1). Then, the 3D points are projected onto the 2D image coordinates through the camera intrinsic parameters, so that the position of the corresponding point can be found in the image; Step 2. Semantic segmentation result and field of view (FOV) constraint: Given an image S (each pixel of which represents the binary result of whether it is drivable or not) (i.e., the first 2D image) after semantic segmentation, median filtering and morphological closing operation are first performed on it to obtain a more continuous and smooth drivable area mask; then, the FOV polygon in the 3D space is calculated according to the horizontal and vertical field of view (FOV) of the camera, and it is projected onto the image plane to obtain the FOV polygon region F in the image coordinate system (i.e., the projected polygon region). Only those falling into F Points within the area are then considered further; Step 3. Point cloud classification and processing within the area: For laser points projected onto the image, when the image pixel corresponding to pimg (i.e., the projected point cloud data) is a drivable area in S (pixel value > 0), the 3D point is marked as "internal drivable point".

[0075] If the corresponding pixel is a non-drivable region in S (pixel value = 0), then try to find the nearest drivable neighboring pixel in the closed mask (i.e., the second 2D image), and mark the corresponding 3D point as "boundary points"; using the above method, a set of points can be obtained, as shown in the following formula (3); Step 4. Drivable region construction and detection box filtering: (i.e., the set of internal drivable points) and (i.e., the boundary point set) are packaged into AreaPoints information and stored in the DrivableSpace message to form a basic drivable area description, as shown in the above formula (2). Then, based on the identified target detection boxes (the result of 3D radar detection), these area points are filtered to remove area points with excessively high overlap ratios in the detection boxes, thereby obtaining the filtered point set: A_filtered A; This step is equivalent to selectively filtering the set of drivable points in the semantic space to ensure that the final region better matches the spatial layout of the scene surrounding the vehicle.

[0076] (3) in, Represents the set of internal drivable points. Represents the set of boundary points.

[0077] This embodiment of the application projects 3D point cloud data onto an image plane to obtain projected point cloud data; then, by combining semantically segmented 2D image information (i.e., the first 2D image), drivability is determined from the point cloud data, thereby achieving the fusion processing of 2D and 3D information. This method effectively compensates for the lack of depth information in 2D images and the lack of semantic understanding in 3D point clouds, improving the accuracy and robustness of drivable area recognition while reducing hardware costs and computational complexity.

[0078] Based on the autonomous driving method proposed in the foregoing embodiments, this application also proposes an autonomous driving device, such as... Figure 3 As shown, Figure 3 A schematic diagram of an autonomous driving device is shown, the device comprising: The acquisition module 401 is used to acquire three-dimensional 3D point cloud data and project the 3D point cloud data onto an image plane to obtain projected point cloud data; and to acquire a first 2D image; wherein each pixel in the first 2D image is marked as a drivable area or a non-drivable area. The judgment module 402 is used to determine whether the projected point cloud data is a drivable area based on the first 2D image; If the construction module 403 is a drivable area, it constructs a target drivable area based on the projected point cloud data, so as to perform autonomous driving based on the target drivable area.

[0079] In practical applications, the acquisition module 401, the judgment module 402, and the construction module 403 can be implemented based on a processor and a communication device.

[0080] In some embodiments, the acquisition module 401 is specifically used to transform the 3D point cloud data using a first transformation matrix to obtain 3D points in the camera coordinate system; and to project the 3D points in the camera coordinate system using a second transformation matrix to obtain the projected point cloud data.

[0081] In some embodiments, the determination module 402 is specifically used to determine a polygonal region in 3D space based on the horizontal and vertical field of view of the camera mounted on the vehicle; project the polygonal region onto an image plane to obtain a projected polygonal region; and, if the first 2D image is located within the projected polygonal region, determine whether the projected point cloud data is a drivable area based on the first 2D image.

[0082] In some embodiments, the determination module 402 is further configured to perform preset processing on the first 2D image to obtain a processed first 2D image; wherein, the preset processing includes median filtering and morphological closing operation processing; determine whether the projected point cloud data is a drivable area in the processed first 2D image; if the projected point cloud data is a drivable area, then mark the projected point cloud data as a drivable point.

[0083] In some embodiments, the construction module 403 is configured to acquire a second 2D image; wherein the second 2D image includes the first 2D image; query the drivable neighboring pixels that are closest to the non-drivable point in the second 2D image; and construct the target drivable area based on the neighboring pixels and the drivable point.

[0084] In some embodiments, the construction module 403 is further configured to mark the 3D point cloud data corresponding to the neighboring pixels as boundary points; and construct the target drivable area based on the boundary points and the 3D point cloud data corresponding to the drivable points.

[0085] It should be noted that the description of the apparatus in this application is similar to the description of the method embodiments described above, and has similar beneficial effects as the same method embodiments, so it will not be repeated.

[0086] This application also provides an electronic device. Figure 4 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, as shown below. Figure 4 As shown, electronic device 500 may include at least one processor 510, memory 540, and at least one network interface 520. The various components in electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to implement communication between these components. In addition to a data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general labeled all buses as Bus System 530.

[0087] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0088] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 540 may optionally include one or more storage devices physically located away from the processor 510.

[0089] The memory 540 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 540 described in this application embodiment is intended to include any suitable type of memory.

[0090] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0091] Operating system 541 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0092] The network communication module 542 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0093] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 4 An autonomous driving device 40 stored in memory 540 is shown. This device can be software in the form of programs and plugins, including the following software modules: an acquisition module 401, a judgment module 402, and a construction module 403. These modules are logically connected and can therefore be arbitrarily combined or further separated according to the functions implemented. It should be noted that... Figure 4 For ease of explanation, all the above modules are shown at once, but this should not be regarded as excluding the implementation of the autonomous driving device 40 which may only include the acquisition module 401, the judgment module 402, and the construction module 403.

[0094] In some embodiments, the computer-readable storage medium may be a memory such as ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0095] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0096] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0097] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0098] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the autonomous driving method described above in this application.

[0099] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0100] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0101] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0102] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0103] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0105] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An autonomous driving method, characterized in that, The method includes: Acquire 3D point cloud data and project the 3D point cloud data onto an image plane to obtain projected point cloud data; Acquire a first 2D image; wherein each pixel in the first 2D image is marked as a drivable area or a non-drivable area; Based on the first 2D image, it is determined whether the projected point cloud data is a drivable area; if it is a drivable area, a target drivable area is constructed based on the projected point cloud data, so as to perform autonomous driving of the vehicle based on the target drivable area.

2. The method according to claim 1, characterized in that, After acquiring the first 2D image, the method further includes: The polygonal region in 3D space is determined based on the horizontal and vertical field of view of the camera mounted on the vehicle. The polygonal region is projected onto the image plane to obtain the projected polygonal region; If the first 2D image is located within the polygonal region after projection, determine whether the projected point cloud data is a drivable area based on the first 2D image.

3. The method according to claim 2, characterized in that, The step of determining whether the projected point cloud data is a drivable area based on the first 2D image includes: The first 2D image is subjected to a preset processing to obtain a processed first 2D image; wherein, the preset processing includes median filtering and morphological closing operation processing; Determine whether the projected point cloud data is a drivable area in the processed first 2D image; If the projected point cloud data is a drivable area, then the projected point cloud data is marked as a drivable point.

4. The method according to claim 3, characterized in that, The method further includes: If the projected point cloud data is not a drivable area, then the projected point cloud data will be marked as a non-drivable point.

5. The method according to claim 3 or 4, characterized in that, The construction of the target drivable area based on the projected point cloud data includes: Acquire a second 2D image; wherein the second 2D image contains the first 2D image; In the second 2D image, query the nearest drivable neighboring pixel to the undrivable point; The target drivable area is constructed based on the neighboring pixels and drivable points.

6. The method according to claim 5, characterized in that, The construction of the target drivable region based on the neighboring pixels and drivable points includes: Mark the 3D point cloud data corresponding to the neighboring pixels as boundary points; The target drivable area is constructed based on the 3D point cloud data corresponding to the boundary points and the drivable points.

7. The method according to claim 1, characterized in that, The step of projecting the 3D point cloud data onto an image plane to obtain the projected point cloud data includes: The 3D point cloud data is transformed using the first transformation matrix to obtain 3D points in the camera coordinate system; The 3D points in the camera coordinate system are projected using the second transformation matrix to obtain the projected point cloud data.

8. An automatic driving device, characterized in that, The device includes: The acquisition module is used to acquire three-dimensional 3D point cloud data and project the 3D point cloud data onto an image plane to obtain projected point cloud data; and to acquire a first 2D image; wherein each pixel in the first 2D image is marked as a drivable area or a non-drivable area; The judgment module is used to determine whether the projected point cloud data is a drivable area based on the first 2D image; If the construction module is a drivable area, it constructs a target drivable area based on the projected point cloud data, and performs autonomous driving based on the target drivable area.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the autonomous driving method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or computer-executable instructions stored thereon, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the autonomous driving method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the autonomous driving method according to any one of claims 1 to 7 is implemented.

Images