Ground sign segmentation method, vehicle and medium

Abstract

This application discloses a ground marking segmentation method, a vehicle, and a medium for use in a vehicle. The method includes: acquiring a current scene image captured by a camera on the vehicle; and processing the scene image to generate a bird's-eye view, the bird's-eye view including ground area markings and ground markers of the current scene. By acquiring the current scene image captured by the vehicle's camera and processing the scene image to generate a bird's-eye view, which includes ground area markings and ground markers of the current scene, the bird's-eye view can be accurately segmented from the ground area to the non-ground area and ground markers (such as lane lines, driving arrows, and text markings) can be identified, avoiding misidentification and missed identification of markings, and enabling the vehicle to recognize driving-related markings and driving surfaces.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and more specifically, to a ground marking segmentation method, vehicles, and a non-volatile computer-readable storage medium. Background Technology

[0002] In related technologies, with the development of autonomous driving technology, the vehicle's accurate perception of the road environment has become a crucial element in ensuring driving safety and the effectiveness of decision-making. During vehicle operation, the ability of the vehicle to recognize driving-related signs is a problem that urgently needs to be solved in this field. Summary of the Invention

[0003] This application provides a ground marking segmentation method, a vehicle and a non-volatile computer-readable storage medium, which can solve at least one of the above technical problems.

[0004] In a first aspect, embodiments of this application provide a ground marking segmentation method for vehicles, the method comprising: Acquire the current scene image captured by the vehicle's camera; The scene image is processed to generate a bird's-eye view, which includes ground area markers and ground markings for the current scene.

[0005] Secondly, embodiments of this application provide a vehicle, including: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the ground marker segmentation method described in any of the above embodiments.

[0006] Thirdly, embodiments of this application provide a non-volatile computer-readable storage medium containing a computer program, which, when executed by a processor, causes the processor to perform the ground marker segmentation method described in any of the above embodiments.

[0007] This application describes a ground marking segmentation method, a vehicle, and a medium. By acquiring current scene images captured by the vehicle's camera, the scene images are processed to generate a bird's-eye view. The bird's-eye view includes ground area markings and ground markings in the current scene. By generating a bird's-eye view based on the scene images, ground areas and non-ground areas can be accurately segmented, and ground markings (such as lane lines, driving arrows, and text markings) can be identified, avoiding misidentification or omission of markings, and enabling the vehicle to recognize driving-related markings and driving surfaces.

[0008] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0009] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein: Figure 1 This is a schematic diagram illustrating an application scenario of the ground marking segmentation method according to certain embodiments of this application; Figure 2 This is a flowchart illustrating a ground marking segmentation method according to certain embodiments of this application; Figures 3-10 This is a schematic diagram of a scenario for a ground marking segmentation method according to certain embodiments of this application; Figure 11 This is a schematic diagram of a ground marking segmentation device according to certain embodiments of this application; Figure 12 This is a schematic diagram illustrating the connection state of a non-volatile computer-readable storage medium and a processor in certain embodiments of this application. Detailed Implementation

[0010] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0011] To facilitate understanding of this application, the following explanations are provided for the terms and background information used in this application: In related technologies, when training models to generate Bird's-Eye-View (BEV) images, the BEV base map used for annotation is typically obtained by stitching together multiple frame point clouds. Lidar intensity values ​​(which can be used to distinguish different target types or obstacles) are then filled into the image within the vehicle's BEV image coordinate system. Since the stitched point cloud represents the vehicle's global trajectory, blind spots are minimal. However, for training data from single-frame images or point clouds, the sensor's field of view is limited, and due to obstacle occlusion, only partial areas can be perceived. For example, due to wall occlusion, markings invisible to the vehicle become visible because the BEV base map is a stitched-together multi-frame point cloud, increasing unnecessary computation.

[0012] For BEV ground marking training data used in the training process, annotation is usually divided into manual annotation and semi-automatic annotation. Manual annotation can indicate the visibility of parking spaces, but pixel-level annotations such as lane lines are not visible. Furthermore, the accuracy of manual annotation is difficult to guarantee, resulting in high cost and low efficiency. Semi-automatic annotation uses ground lines and shadow cancellation lines (or segments a large model to generate ground lines) to generate visibility. Its advantage is that ground line annotation is very accurate, but its disadvantages include the need to annotate ground lines, the inability to distinguish vehicle bodies, significant errors, and the lack of height information at ground lines, making it impossible to determine the ground beyond the ground lines. Both manual and semi-automatic annotation have efficiency and accuracy issues. Additionally, due to occlusion by obstacles such as walls, some parking spaces may be in the blind spot of the current frame image, requiring further processing.

[0013] In view of this, this application provides a method for ground marking segmentation, a method for generating a bird's-eye view, and a vehicle. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram illustrates an application scenario for an embodiment of this application. The ground marking segmentation method can be executed by a vehicle 100; alternatively, the ground marking segmentation method can also be executed by a computer device, which may be a terminal 200 or a server, etc.

[0014] The vehicle includes at least one camera, which is mounted on the vehicle body to capture images of the scene around the vehicle. For example, the vehicle may include four cameras, which are respectively mounted on the front, rear and sides of the vehicle body.

[0015] Alternatively, the camera can be a visible light camera (Red-Green-Blue, RGB), a visible light depth camera (Red-Green-Blue-Depth, RGBD), etc. RGB cameras and RGBD cameras can capture visible light images of the scene.

[0016] The vehicle also includes one or more radar sensors, which are used to collect point cloud information of objects in the scene. The radar sensors can be laser radar (LDS), such as TOF radar based on the time-of-flight (TOF) principle, or structured light radar based on the structured light principle.

[0017] Please see Figure 2 and Figure 3 This application also proposes a ground marking segmentation method, which includes: Step 011: Obtain the current scene image captured by the vehicle's camera.

[0018] The vehicle can capture scene images of the current environment using at least one camera. These images can be two-dimensional images of the surrounding environment, reflecting its visual texture. This can be either real-time images of the current scene during the vehicle's movement, or images of scenes the vehicle has experienced during its journey (not necessarily real-time) captured by the vehicle's cameras for subsequent processing.

[0019] Step 012: Process the scene image to generate a bird's-eye view, which includes ground area markers and ground markings for the current scene.

[0020] Ground area markers can be used to mark ground areas in a bird's-eye view, and ground marking markers can be used to mark ground markings in a bird's-eye view (such as lane lines, parking spaces, stop lines, etc.). Specifically, the scene image of the current scene can be acquired first by the vehicle's camera, and then processed by a preset ground segmentation model to obtain the current bird's-eye view of the current scene. The bird's-eye view includes ground area identifiers and ground marker identifiers of the current scene to distinguish the ground areas and ground markers in the scene.

[0021] In this way, by acquiring the current scene image captured by the vehicle's camera, the scene image is processed to generate a bird's-eye view. The bird's-eye view includes ground area markers and ground mark markers of the current scene, which can distinguish the ground areas and ground marks in the scene and improve applicability.

[0022] Optionally, the transformation of the reference position and / or pose of each point cloud data is obtained based on a preset matrix calculation.

[0023] In the embodiments of this application, for ease of explanation, when the point cloud data uses the vehicle position as the reference position, it is considered to be point cloud data in the vehicle coordinate system; when the point cloud data uses the vehicle position in the bird's-eye view as the reference position, it is considered to be point cloud data in the bird's-eye view coordinate system; when the point cloud data uses the global position (world physical position) as the reference position, it is considered to be point cloud data in the world coordinate system; and when the point cloud data uses the radar sensor position as the reference position, it is considered to be point cloud data in the radar coordinate system.

[0024] In some implementations, the method further includes: Step 013: Acquire point cloud data collected by the radar sensor; Step 014: Generate a bird's-eye view of the visible area of ​​the current scene. The visible area is determined based on the ground area and obstacle positions determined by point cloud data, as well as camera parameters.

[0025] Among them, the point cloud data collected by radar sensors can reflect the three-dimensional position of the ground and obstacles (walls, vehicles, etc.) in the environment.

[0026] Among them, the bird's-eye view of the visible area (BEV visibility base map) can be an image of the ground area that the camera can actually observe, calculated from point cloud data and camera parameters in the vehicle BEV (bird's-eye view) coordinate system. The image includes a distinction between visible and invisible areas (occluded or outside the field of view).

[0027] The ground region can be a spatial region that represents the ground in the current scene.

[0028] The obstacle location can be the location of an obstacle existing in the current scene, which can be identified through point cloud data.

[0029] Camera parameters can include intrinsic parameters (such as focal length and pixel size) and extrinsic parameters.

[0030] Specifically, based on the three-dimensional spatial information of point cloud data (ground area, obstacle location, obstacle height, etc.) and camera parameters, ground areas that are unobstructed and within the field of view of the camera can be filtered out through methods such as coordinate transformation and geometric calculations, and the area that the vehicle can perceive can be determined, thereby summarizing to obtain a bird's-eye view.

[0031] Please see Figure 4 In some embodiments, the ground marking segmentation method further includes: Step 015: Acquire multiple consecutive frames of point cloud data, with the vehicle position as the reference position for the point cloud data; Step 016: Convert the relative reference positions of consecutive multi-frame point cloud data to obtain consecutive multi-frame world point cloud data; Step 017: Erase the world point cloud data according to the preset erasure area range, and stitch together the erased world point cloud data of multiple consecutive frames to obtain globally optimized point cloud data; Optionally, the global optimized point cloud data can be calculated based on the preset second transformation matrix (T_lidar2ego) and the first transformation matrix to obtain the third transformation matrix (T_world2ego). The second transformation matrix includes the transformation relationship between the radar coordinate system and the vehicle coordinate system, and the third transformation matrix includes the transformation relationship between the world coordinate system and the vehicle coordinate system.

[0032] Optionally, a first transformation matrix (T_lidar2world) can be obtained first. The first transformation matrix (T_lidar2world) can be a pre-calibrated transformation matrix between the radar coordinate system and the world coordinate system, which can be used to describe the position and attitude mapping relationship between the radar coordinate system and the world coordinate system under the radar sensor.

[0033] The world coordinate system can be a three-dimensional reference coordinate system for the current scene. It can be used to unify the spatial positions of multi-frame point cloud data, vehicles, radar sensors, and cameras, ensuring that multi-frame data are stitched and processed under the same reference coordinate system.

[0034] The erased area can be the spatial range of the vehicle body in the world coordinate system (for example, it can be obtained by mapping the vehicle body range in the vehicle coordinate system through a transformation matrix). It is used to remove point clouds belonging to the vehicle itself from the point cloud data to avoid the vehicle's own point cloud data from interfering with subsequent obstacle recognition.

[0035] Among them, the globally optimized point cloud data can be the point cloud data after erasing the erased area and the point cloud data set after stitching together multiple frames.

[0036] The second transformation matrix (T_lidar2ego) can be a transformation matrix between the pre-calibrated radar coordinate system and the vehicle coordinate system.

[0037] The third transformation matrix (T_world2ego) can be the transformation matrix between the world coordinate system and the vehicle coordinate system, which is calculated using the second transformation matrix (T_lidar2ego) and the first transformation matrix (T_lidar2world).

[0038] Specifically, due to the limited field of view and incomplete environmental information of single-frame radar point cloud data (e.g., inability to cover the entire surrounding environment of the vehicle), multi-frame point cloud data can be stitched together to integrate environmental information from multiple frames, compensating for the field of view deficiencies of a single frame and obtaining a globally complete environmental point cloud. Furthermore, the point cloud data collected by radar sensors typically includes the vehicle's own point cloud data, which can interfere with the subsequent distinction between the ground and obstacles. Therefore, invalid point clouds need to be erased according to a preset erasure area (corresponding to the vehicle's own area) to improve data purity. Additionally, to avoid acquiring data based on different coordinate systems, The issue of coordinate system confusion causing errors necessitates coordinate system transformation of the data to ensure all data is processed under a unified benchmark. Therefore, this can be achieved by acquiring continuous raw multi-frame point cloud data and using a first transformation matrix (T_lidar2world) to transform the continuous multi-frame point cloud data from the radar (lidar) coordinate system to the world coordinate system, obtaining multi-frame world point cloud data in the world coordinate system. Then, according to a preset erasure area, the world point cloud data is erased, and the erased world point cloud data from multiple consecutive frames is stitched together to obtain globally optimized point cloud data (i.e.,...). Figure 4The global point cloud (in the image) needs to be denoised after the multi-frame world point cloud data is stitched together to remove discrete noise points, further improve the data quality, and provide reliable input for subsequent processing. Finally, the transformation matrix (third transformation matrix) from the global optimized point cloud data to the vehicle coordinate system is calculated by using T_world2ego=T_lidar2ego*inv(T_lidar2world) based on the first transformation matrix (T_lidar2world) and the preset second transformation matrix (T_lidar2ego).

[0039] Please see Figure 5 Optionally, step 014: Generate a bird's-eye view of the visible area of ​​the current scene, including: Step 0141: Based on the bird's-eye view, crop the globally optimized point cloud data to obtain the effective point cloud data for the vehicle; Step 0142: Perform ground plane fitting on the effective point cloud data of the vehicle to obtain ground plane parameters, which are used to describe the spatial morphology of the ground. Step 0143: Based on the ground plane parameters, filter the ground point cloud data and obstacle point cloud data, and generate the first bird's-eye view base map based on the filtering results and the preset bird's-eye view base map; Step 0144: Divide the first bird's-eye view base map (obstacle height base map) according to the parameters of each camera, and merge the divided results to generate a bird's-eye view of the visible area of ​​the current scene.

[0040] The bird's-eye view range can be a preset effective perception range around the vehicle. For example, it can be the boundary range of the bird's-eye view (BEV base map) generated with the vehicle as the center, with ±12 meters in front and behind the vehicle, ±12 meters to the left and right of the vehicle, and a height of 0.7 to 1.8 meters.

[0041] Among them, the effective point cloud data of the vehicle can be point cloud data that is cropped from the global optimized point cloud and located within the bird's-eye view range, retaining only the environmental point cloud related to the vehicle and removing redundant data.

[0042] Regarding ground plane parameters, these parameters can be ground plane equations. Optionally, step 0142 involves fitting the ground plane to the effective point cloud data of the vehicle to obtain the ground plane parameters, including: Step 01421: Filter out ground point cloud data from the valid point cloud data of vehicles, and perform ground plane fitting on the ground point cloud data to obtain ground plane parameters, which can be ground plane equations.

[0043] Among them, the ground plane equation can be obtained by filtering out suspected ground point cloud data from the effective point cloud data of vehicles through algorithms such as RANSAC, and then performing plane fitting on the filtered point cloud data to obtain a mathematical equation that can describe the spatial morphology of the ground (such as ax+by+cz+d=0).

[0044] Among them, ground point cloud data can be point cloud data whose distance from the ground plane equation is less than a preset threshold (such as 0.1 meters, 0.2 meters, etc.), which can reflect the spatial position of the real ground; obstacle point cloud data can be point cloud data whose distance from the ground plane equation is greater than a preset threshold, corresponding to the spatial position of obstacles such as walls, vehicles, and trees in the scene.

[0045] The first bird's-eye view base map (obstacle height base map) can be a preset BEV base map (assuming a resolution of 1280×1280) as a carrier, and the image generated after mapping the height values ​​of the obstacle point cloud to grayscale values ​​(the higher the grayscale, the higher the obstacle height).

[0046] Optionally, step 0141: Based on the bird's-eye view, the globally optimized point cloud data is cropped to obtain effective point cloud data for the vehicle. Step 01411: Based on the preset bird's-eye view range, the global optimized point cloud data is cropped to obtain the global effective point cloud data. The bird's-eye view range is obtained by connecting multiple global corner coordinates in sequence, and the global corner coordinates are matched with the corners of the vehicle. Step 01412: Convert the baseline position of the global effective point cloud data into the vehicle position, and trim the converted global effective point cloud data according to the bird's-eye view area to obtain the vehicle effective point cloud data.

[0047] For example, the third transformation matrix (T_world2ego) can be used to transform the corner coordinates of multiple vehicle coordinate systems to the world coordinate system to obtain multiple global corner coordinates. The bird's-eye view area and the bird's-eye view range formed by connecting the multiple corner coordinates in sequence are matched. Then, based on the bird's-eye view range, the global optimized point cloud data is cropped to obtain the global effective point cloud data in the world coordinate system. The bird's-eye view range is obtained by connecting the multiple global corner coordinates in sequence. Next, the third transformation matrix (T_world2ego) is used to transform the global effective point cloud data to the vehicle coordinate system, and the transformed global effective point cloud data is pruned according to the bird's-eye view area to obtain the vehicle effective point cloud data.

[0048] First, the corner coordinates in multiple vehicle coordinate systems can be obtained, for example, [[-12,-12],[-12,12],[12,-12],[12,12]]. The resulting bird's-eye view area and range are matched by connecting these coordinates sequentially. Then, the coordinates of these intersection points are transformed to the world coordinate system using a third transformation matrix (T_world2ego), yielding the boundary values ​​of the BEV image in the world coordinate system. Finally, the boundary values ​​(and...) are used to... Figure 5 The global optimized point cloud data is cropped according to the vehicle range in the world coordinate system to obtain the global effective point cloud data. Then, the global effective point cloud data is transformed to the vehicle coordinate system through the third transformation matrix (T_world2ego). The transformed global effective point cloud data is further trimmed according to the bird's-eye view area to obtain the vehicle effective point cloud data (single frame vehicle point cloud).

[0049] Optionally, step 0143: Based on ground planar parameters, filter ground point cloud data and obstacle point cloud data, and based on the filtering results and a preset bird's-eye view base map, generate a first bird's-eye view base map, including: Step 01431: Filter obstacle point cloud data based on the distance between the effective point cloud data of each vehicle and the ground plane parameters; Step 01432: Transform the reference position of the obstacle point cloud data so that the transformed bird's-eye view obstacle point cloud data is visible in the current bird's-eye view; Step 01433: Subtract a preset height threshold from the height values ​​of the bird's-eye view obstacle point cloud data, and multiply the subtraction result by a preset height scaling factor. Use the multiplication result as the pixel grayscale value to fill the preset bird's-eye view base map to generate the first bird's-eye view base map. The preset height threshold is determined based on the minimum height value of the bird's-eye view area. When multiple point clouds are projected onto the same base map pixel, the largest height value is retained. Finally, the obstacle height base map is obtained.

[0050] Optionally, the fourth transformation matrix (T_ego2img) can be a transformation matrix between the vehicle coordinate system and the bird's-eye view coordinate system, calculated based on the bird's-eye view area and the preset bird's-eye view base map size (such as 1280×1280 resolution).

[0051] The preset height threshold can be the minimum height value (Z_min) determined based on the height values ​​of all point clouds in the bird's-eye view area. It can be used to unify the calculation benchmark for obstacle height and ensure the rationality of grayscale value mapping.

[0052] The height scaling factor (height_scale) can be a preset fixed coefficient used to map the obstacle height difference (Z-Z_min) to a grayscale value range (such as 0 to 255) to adapt to the grayscale range of image pixels and realize the conversion between height information and grayscale information.

[0053] First, obstacle point cloud data can be filtered based on the ground plane equation and the vehicle's effective point cloud data. Then, the fourth transformation matrix (T_ego2img) can be calculated based on the bird's-eye view area and the preset bird's-eye view base map size (such as 1280×1280 resolution). The obstacle point cloud data can be transformed from the vehicle coordinate system to the bird's-eye view coordinate system through the fourth transformation matrix (T_ego2img) to obtain the bird's-eye view obstacle point cloud data in the bird's-eye view coordinate system (including the pixel coordinates of each obstacle point cloud data in the bird's-eye view map). Next, iterate through all obstacle point cloud data. Based on the height Z value of each obstacle point cloud data, calculate the minimum height value Z_min within the bird's-eye view range. For example, calculate the grayscale value using the formula: grayscale value = (Z - Z_min) × height_scale (height_scale is a preset height scaling factor). Finally, fill the grayscale value into the corresponding pixel coordinates of the BEV base map. After filling in the height values ​​of all pixels, you will obtain the first bird's-eye view base map with a resolution of 1280×1280 (a higher grayscale value indicates a higher obstacle height). Figure 5 In the obstacle height base map, it should be noted that if multiple obstacle point cloud data are projected onto the same pixel, the maximum gray value is retained as the gray value of that pixel.

[0054] Specifically, since the global optimization point cloud data has a wide coverage area and contains a large amount of redundant point cloud data that is irrelevant to the vehicle perception of the current frame, it can be cropped according to the bird's-eye view range to reduce the amount of subsequent calculations and improve processing efficiency. Therefore, based on the bird's-eye view range, the globally optimized point cloud data can be cropped into point cloud data within the bird's-eye view range that are related to the vehicle's surroundings, i.e., effective vehicle point cloud data. Then, the effective vehicle point cloud data can be filtered using algorithms such as RANSAC to remove outliers that are significantly deviated from the ground (such as high-altitude suspended objects, isolated noise points, etc.), resulting in a set of suspected ground point clouds. Spatial plane fitting is then performed on the suspected ground point cloud set, and the ground plane equation ax + by + cz + d = 0 is obtained by solving the equation using methods such as least squares (a, b, c, d are fitting coefficients that reflect the tilt angle and spatial position of the ground). Next, each point cloud in the effective vehicle point cloud is traversed, and the distance from each point cloud to the ground plane equation is calculated. Based on the distance, the ground point cloud data and obstacle point cloud data are determined. Then, the preset bird's-eye view base map is processed according to the filtering results to obtain the first bird's-eye view. Finally, the first bird's-eye view base map is divided according to the parameters of each camera to obtain multiple divided base maps. The multiple divided base maps are then merged to generate a bird's-eye view of the visible area of ​​the current scene.

[0055] Please see Figure 6 and Figure 7 Optionally, step 0144: Based on the parameters of each camera, the first bird's-eye view base map (obstacle height base map) is divided into segments, and the segmented results are merged to generate a bird's-eye view of the visible area of ​​the current scene, including: Step 01441: Initialize the bird's-eye view base map based on the base maps of multiple viewpoint regions. The base map of the field of view region is determined based on the camera parameters. Step 01442: Extract connected regions from the first bird's-eye view base map to determine the occluded areas; Step 01443: Based on the occlusion area, divide the base map of each initialized view area to obtain multiple visible area base maps; Step 01444: Perform an OR operation on multiple visible area base maps to generate a bird's-eye view of the visible areas of the current scene.

[0056] Specifically, please refer to Figure 6 Taking a vehicle with four cameras as an example, the process can be explained as follows: First, initialize the bird's-eye view base map based on the base map of the viewing area corresponding to each camera, setting the pixels outside the camera's field of view to 0. Then, extract the connected components of the first bird's-eye view base map using 8-connected regions to determine the occlusion areas that can be blocked by obstacles. Next, mark the areas corresponding to the occlusion areas as invisible areas on each initialized viewing area base map. Removing the occlusion areas from the viewing area base map yields the visible area base map. For example, please refer to [link to relevant documentation]. Figure 6, Figure 6 An example is shown, illustrating the field of view (FOV) of four cameras. The white areas represent the visible areas from the camera's perspective. Please refer to [link / reference]. Figure 7 Multiple visible area base maps are combined into a single visible base map through an OR operation (i.e., for a certain area, if it is visible on any visible area base map, then the area is considered to be in the visible area).

[0057] Optionally, the camera parameters include camera field of view parameters, and also include: Step 018: Obtain the poses of each camera in the vehicle coordinate system; Step 019: Perform pose transformation on each camera so that the transformed camera poses are visible in the current bird's-eye view (i.e., transform to the bird's-eye view coordinate system). Step 020: Generate multiple view area base maps based on the converted camera poses and corresponding camera field of view parameters. The camera poses include the camera coordinates and orientation angle.

[0058] Specifically, the poses of each camera, P_cam, can be initialized first (P_cam is a 4x4 identity matrix). Then, according to the fifth transformation matrix (T_cam2ego, the transformation matrix between the camera coordinate system and the vehicle coordinate system), the poses of each camera are transformed to the vehicle coordinate system. The poses in the vehicle coordinate system are P_ego = T_cam2ego * P_cam. Then, through the fourth transformation matrix, the camera poses are transformed to the bird's-eye view coordinate system, so that the transformed poses of each camera are visible in the current bird's-eye view. The transformed poses are P_img = T_ego2img * P_ego. The coordinates and orientation angle of the camera in the image are extracted from the P_img matrix, and the FOV angle parameters of the camera are combined to generate the view area base map (FOV area map) of each camera.

[0059] Optionally, step 01442: Extract connected components from the first bird's-eye view base map to determine occluded areas. Step 01451: Denoise the first bird's-eye view base image to obtain the outline of the obstacles; Step 01452: Extract 8-connected components from the obstacle contour to determine at least one obstacle connected component; Step 01453: Extract the contour corner points of each obstacle's connected domain, and determine the occlusion area based on the camera pose, ground plane equation, and contour corner points.

[0060] Specifically, the first bird's-eye view can be denoised and obstacle edges completed using image morphology methods to obtain obstacle contours. Then, 8-connected component extraction is performed on the obstacle contours to determine at least one obstacle connected component (used to represent obstacles in the scene). Next, the contour corner points of each obstacle connected component are extracted, and the occlusion area is determined based on the camera pose, ground plane equation, and contour corner points. For example, please refer to [link to relevant documentation]. Figure 8 Extract the coordinates P_k of the k-th corner point, calculate the ray connecting the corner point and the camera, and calculate the intersection point P_m of this ray with the ground plane equation (if the intersection point is outside the base map or has no intersection with the plane, the intersection point is the projection of the ray onto the boundary of the base map); extract the coordinates P_k+1 of the (k+1)-th corner point, and similarly calculate the intersection point P_m+1 (if k+1 ≥ the total number of corner points, then extract the 0th corner point); the four points (P_k, P_k+1, P_m+1, P_m) form a polygonal region, which can be filled using the cv2.fillPoly() function. This region is the area invisible to the camera, i.e., the occluded region.

[0061] In some implementations, step 012: processing the scene image to generate a bird's-eye view includes: Step 0121: Process the scene image using a preset ground segmentation model (i.e., BEV ground marking segmentation model) to generate a bird's-eye view. The ground segmentation model is trained on the initial model based on the scene image, the first label image, and the second label image. The first label image is generated based on the ground area in the bird's-eye view and the scene image, and the second label image is generated based on the bird's-eye view and the preset label image containing ground markings.

[0062] Optionally, the ground region in the bird's-eye view can be projected onto each scene image to generate a first label image corresponding to the ground region in each scene image; a preset label image containing ground markings can be processed based on the bird's-eye view to generate a second label image containing the visible area; and an initial model can be trained based on the scene image, the first label image, and the second label image to generate a ground segmentation model.

[0063] The first label image can be a ground region image generated by projecting the ground region from the bird's-eye view onto a two-dimensional scene image, and can be used as a supervision label for the ground segmentation task (the ability to identify ground regions) during model training.

[0064] In the BEV visibility base map, the ground region is the real visible ground in three-dimensional space. It can be mapped onto the scene image of the two-dimensional camera through coordinate inverse transformation and camera projection principle to obtain a label for whether each pixel in the scene image is a visible ground. The label can be used to supervise the ground segmentation model to learn the ability to identify ground regions from the camera image, thereby overcoming the problems of low efficiency and poor consistency of manual annotation of ground regions.

[0065] The preset label image can be an original labeled image containing ground markings (such as lane lines, parking spaces, stop lines, etc.) (which can be manually or semi-automatically labeled BEV base map labels).

[0066] The second label image can be a label image that retains only the ground markings in the visible area after filtering the bird's-eye view through preset label images. It can also be a supervision label for the ground marking segmentation task (learning the ability to identify specific markings from the ground area) in the training of the ground segmentation model.

[0067] The preset label image contains ground markings for both visible and invisible areas, but the ground segmentation model only needs to learn the ground markings for the visible areas (the markings for the invisible areas are meaningless for the current frame). A second label image can be generated by using a bird's-eye view of the visible areas of the current scene as a filter mask to remove invalid labels from the invisible areas, thus avoiding the ground segmentation model learning noisy data. Simultaneously, the invisible areas are set as negative samples, optimizing the sample distribution.

[0068] The initial model can be an untrained neural network model.

[0069] The ground segmentation model can be a model that, after being trained under dual supervision by the first label image and the second label image, can simultaneously output the segmentation results of the ground region and the ground label.

[0070] Specifically, the initial model can be trained by using the first labeled image as supervision to learn the ability to identify ground regions from camera images, and by using the second labeled image as supervision to learn the ability to identify specific markers from ground regions. The model parameters can be adjusted by combining the loss values ​​corresponding to the two labeled images to train the initial model until the loss value reaches the preset loss threshold condition, thereby obtaining a ground segmentation model that can simultaneously output ground regions and ground markers.

[0071] Optionally, the camera parameters include the image transformation relationship between the vehicle coordinate system and the image coordinate system, and the method further includes: Step 021: Obtain the ground visibility parameters and match them with the ground point cloud data; Step 022: Based on the visible ground parameters and the ground plane parameters, determine the height value of the visible ground parameters, and reconstruct the three-dimensional ground parameters corresponding to the visible ground parameters in the space where the vehicle is located using the height value; Step 023: Based on the image transformation relationship, the 3D ground parameters are transformed to the image coordinate system to determine the ground area, and the first label image is generated based on the ground area and the scene image.

[0072] For example, the ground visibility parameters can include the pixel coordinates of the ground visible pixels in the base map of each visible area. The pixel coordinates are transformed to the vehicle coordinate system through the fourth transformation matrix, and the height value is calculated by the ground plane equation using the transformed pixel coordinates. The ground visible pixels and ground point cloud data are matched. Then, based on the transformed pixel coordinates and height values, the three-dimensional ground parameters (i.e., three-dimensional ground coordinates) of the ground visible pixels are restored. Then, based on the image transformation relationship, the three-dimensional ground coordinates are transformed to the image coordinate system to determine the ground area. Based on the ground area and the scene image, the first label image is generated.

[0073] Specifically, please refer to Figure 9 The visible ground pixels P_uv in each visible area base map are transformed to the vehicle coordinate system using the fourth transformation matrix. The transformed pixel coordinates are P_ego_xy = inv(T_ego2img) * P_uv. Then, the pixel coordinates of P_ego_xy are substituted into the ground plane equation to calculate the height value z. Based on the pixel coordinates of P_ego_xy and the height value z, the three-dimensional ground coordinates P_ego_xyz of the visible ground pixels are restored. According to the image transformation relationship between the vehicle coordinate system and the image coordinate system, the three-dimensional ground coordinates P_ego_xyz are transformed to the camera's image coordinate system to determine the ground area (a set of three-dimensional ground coordinates or an area enclosed by three-dimensional ground coordinates). Based on the ground area and the scene image, and through image morphological closing operations, the ground hole areas are filled and completed to generate the first label image.

[0074] In some implementations, the method further includes: Step 024: Based on the bird's-eye view, retain the pixel values ​​of the visible area in the preset label image containing ground markings, and set the pixel values ​​outside the visible area to 0, so as to generate a second label image containing the visible area.

[0075] Specifically, when an aerial view is obtained, the preset label image containing ground markings is filtered based on the aerial view, retaining only the ground markings located within the visible area, and setting the ground markings and pixel values ​​outside the visible area to 0 (i.e., processing them as negative samples), thereby generating a second label image containing the visible area.

[0076] Please see Figure 10In some implementations, the initial model includes a feature encoding module, a ground segmentation module, and a ground identification module, and the method further includes: Step 025: The scene image is feature-encoded using the feature encoding module to generate the first feature vector; Step 026: Calculate the first feature vector using the ground segmentation module to obtain the first predicted image, and calculate the first loss value between the first predicted image and the first label image; Step 027: Calculate the first feature vector using the ground identification module to obtain the second predicted image, and calculate the second loss value between the second predicted image and the second labeled image; Step 028: Adjust the initial model based on the first and second loss values ​​to generate a ground segmentation model.

[0077] Specifically, please refer to Figure 10 First, the image-view encoder performs feature encoding on at least one scene image to generate a first feature vector. This first feature vector is then used by the ground segmentation module (ground segmentation head) to predict ground regions, resulting in a first predicted image of the predicted ground regions. A first label image is used to supervise the first predicted image, and a first loss value is calculated between the first predicted image and the first label image. Next, the view transformation module (view transformer) transforms the first feature vector, and then the BEV encoding module (BEV_encoder) performs feature encoding to obtain a second feature vector. The ground labeling module (BEV ground labeling head) processes the second feature vector to predict ground labeling regions, resulting in a second predicted image. A second label image is used to supervise the second predicted image, and a second loss value is calculated between the second predicted image and the second label image. Based on the sum of the first and second loss values, the initial model is adjusted until the sum of the first and second loss values ​​is less than a preset loss threshold, thus generating a ground segmentation model. Experiments show that compared to ground labeling tasks without visibility processing, the embodiments of this application significantly improve the accuracy and efficiency of this task.

[0078] Thus, by acquiring point cloud data from radar sensors and scene images from cameras, the synchronous acquisition of point cloud data and scene images can ensure the identification error caused by a single sensor and the consistency of subsequent data processing. A bird's-eye view of the visible area of ​​the current scene is generated. The visible area is determined based on the ground area and obstacle positions determined by the point cloud data, as well as camera parameters, which can accurately identify unobstructed visible areas and avoid invalid data interference. The ground area in the bird's-eye view is projected onto each scene image to generate a first label image corresponding to the ground area in each scene image. The first label image can replace manual annotation of ground areas, reducing annotation costs and improving annotation efficiency, enhancing the consistency and accuracy of ground recognition, and improving the accuracy of ground segmentation tasks. Processing of ground markings based on the bird's-eye view is also possible. The system uses preset label images to generate a second label image containing visible areas. The second label image contains ground markings within the visible areas, which reduces noise interference to the model and improves the accuracy of ground marking segmentation. Based on scene images, the first label image, and the second label image, an initial model is trained to generate a ground segmentation model. The entire process, from acquiring point cloud data and scene images, generating bird's-eye view images, generating the first and second label images, to training the initial model to obtain the ground segmentation model, is fully automated without manual intervention, significantly improving training efficiency. Furthermore, the dual-task collaborative training based on the first and second label images can improve model accuracy. The model training is based on point cloud data and scene images collected from real scenes, which can have better generalization performance in complex road environments in practical applications.

[0079] Please see Figure 11 To facilitate better implementation of the ground marking segmentation method of this application, this application also provides a ground marking segmentation device 300. The device 300 includes an acquisition module 301 and a generation module 302. The acquisition module 301 is used to acquire the current scene image captured by the vehicle's camera. The generation module 302 is used to process the scene image to generate a bird's-eye view, which includes ground area markings and ground marker markings of the current scene.

[0080] The apparatus has been described above from the perspective of functional modules in conjunction with the accompanying drawings. These functional modules can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method implementation in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application can be directly manifested as being executed by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps in the above method implementation.

[0081] Please refer to it again. Figure 1 The present application also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, which is executed by the processor. The computer program includes instructions for executing the ground marking segmentation method of any of the above embodiments and can achieve the same technical effect. For the sake of brevity, it will not be described in detail here.

[0082] Please see Figure 1 This application also provides a vehicle, including: A controller, which is used to execute the ground marking segmentation method described in any of the above embodiments.

[0083] Please see Figure 12 This application also provides a computer-readable storage medium 500 storing a computer program 510. When the computer program 510 is executed by the processor 520, it implements the steps of the ground marking segmentation method of any of the above embodiments. For the sake of brevity, these steps will not be repeated here.

[0084] In the description of this specification, the references to terms such as "some embodiments," "in one example," "exemplarily," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0086] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for segmenting ground markings, characterized in that, For use in a vehicle, the method includes: Acquire the current scene image captured by the vehicle's camera; The scene image is processed to generate a bird's-eye view, which includes ground area markers and ground markings for the current scene.

2. The ground marking segmentation method according to claim 1, characterized in that, Also includes: Acquire point cloud data collected by radar sensors; Generate a bird's-eye view of the visible area of ​​the current scene, the visible area being determined based on the ground area and obstacle positions determined by the point cloud data, as well as camera parameters.

3. The ground marking segmentation method according to claim 2, characterized in that, The process of generating a bird's-eye view of the visible area of ​​the current scene includes: Based on the bird's-eye view, the globally optimized point cloud data is cropped to obtain effective point cloud data for the vehicle. Ground plane fitting is performed on the effective point cloud data of the vehicle to obtain ground plane parameters, which are used to describe the spatial morphology of the ground. Based on the ground plane parameters, ground point cloud data and obstacle point cloud data are filtered, and a first bird's-eye view base map is generated based on the filtering results and a preset bird's-eye view base map. Based on the parameters of each camera, the first bird's-eye view is divided into sections, and the results are merged to generate a bird's-eye view of the visible area of ​​the current scene.

4. The ground marking segmentation method according to claim 3, characterized in that, The method further includes: Acquire multiple consecutive frames of point cloud data, with the vehicle position as the reference position; The relative reference positions of the point cloud data in multiple consecutive frames are transformed to obtain world point cloud data in multiple consecutive frames; The world point cloud data is erased according to the preset erasure area range, and the world point cloud data after multiple consecutive erased frames are stitched together to obtain globally optimized point cloud data.

5. The ground marking segmentation method according to claim 4, characterized in that, The step of cropping the globally optimized point cloud data according to the bird's-eye view range to obtain effective vehicle point cloud data includes: According to the preset bird's-eye view range, the global optimized point cloud data is cropped to obtain global effective point cloud data. The bird's-eye view range is obtained by sequentially connecting multiple global corner coordinates, and the global corner coordinates are matched with the corners of the vehicle. The reference position of the global effective point cloud data is converted into the vehicle position, and the converted global effective point cloud data is pruned according to the bird's-eye view area to obtain the vehicle effective point cloud data.

6. The ground marking segmentation method according to claim 2 or 5, characterized in that, The step of filtering obstacle point cloud data based on the ground plane parameters and generating a first bird's-eye view base map based on the obstacle point cloud data and a preset bird's-eye view base map includes: Based on the distance between each vehicle's valid point cloud data and the ground plane parameters, obstacle point cloud data is filtered; The reference position of the obstacle point cloud data is transformed so that the transformed bird's-eye view obstacle point cloud data is visible in the current bird's-eye view; The height values ​​of the bird's-eye view obstacle point cloud data are subtracted from a preset height threshold, and the result of the subtraction is multiplied by a preset height scaling factor. The result of the multiplication is used as the pixel grayscale value and filled into a preset bird's-eye view base map to generate the first bird's-eye view base map. The preset height threshold is determined based on the minimum height value of the bird's-eye view area.

7. The ground marking segmentation method according to claim 3, characterized in that, The step of dividing the first bird's-eye view base map according to the parameters of each camera, and merging the divided results to generate a bird's-eye view of the visible area of ​​the current scene includes: The bird's-eye view base map is initialized based on multiple view area base maps, and the view area base maps are determined based on the camera parameters; Connectivity regions are extracted from the first bird's-eye view base map to determine the occluded areas; Based on the occlusion area, the initial view area base map is divided into multiple visible area base maps. The multiple visible area base maps are ORed to generate a bird's-eye view of the visible area of ​​the current scene.

8. The ground marking segmentation method according to claim 7, characterized in that, The camera parameters include camera field of view parameters, and the method further includes: Obtain the poses of each camera in the vehicle coordinate system; The poses of each camera are transformed so that the transformed poses of each camera are visible in the current bird's-eye view; Based on the converted camera poses and the corresponding camera field of view parameters, multiple view area base maps are generated, wherein the camera poses include the camera coordinates and orientation angle.

9. The ground marking segmentation method according to claim 7, characterized in that, The step of extracting connected regions from the first bird's-eye view base map to determine occluded areas includes: The first bird's-eye view base image is denoised to obtain the outline of the obstacles; The obstacle contour is subjected to 8-connected component extraction to determine at least one obstacle connected component; Extract the contour corner points of each of the connected domains of the obstacles, and determine the occlusion area based on the camera pose, the ground plane parameters, and the contour corner points.

10. The ground marking segmentation method according to any one of claims 3-9, characterized in that, The conversion of the reference position and / or pose of each point cloud data is obtained based on a preset matrix calculation.

11. The ground marking segmentation method according to claim 1, characterized in that, The process of processing the scene image to generate a bird's-eye view includes: The scene image is processed by a preset ground segmentation model to generate a bird's-eye view. The ground segmentation model is obtained by training an initial model based on the scene image, a first label image, and a second label image. The first label image is generated based on the ground area in the bird's-eye view and the scene image, and the second label image is generated based on the bird's-eye view and a preset label image containing ground markings.

12. The ground marking segmentation method according to claim 11, characterized in that, The camera parameters include the image transformation relationship between the vehicle coordinate system and the image coordinate system, and the method further includes: Obtain ground visibility parameters, and match the ground visibility parameters with the ground point cloud data; Based on the ground visibility parameters and ground plane parameters, the height value of the ground visibility parameters is determined, and the three-dimensional ground parameters corresponding to the ground visibility parameters in the space where the vehicle is located are reconstructed using the height value; Based on the image transformation relationship, the three-dimensional ground parameters are transformed into the image coordinate system to determine the ground region, and the first label image is generated based on the ground region and the scene image.

13. The ground marking segmentation method according to claim 11, characterized in that, The method further includes: Based on the bird's-eye view, the pixel values ​​of the preset label image containing ground markings within the visible area are retained, while the pixel values ​​outside the visible area are set to 0, in order to generate a second label image containing the visible area.

14. A vehicle, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the ground marker segmentation method as described in claim 13.

15. A non-volatile computer-readable storage medium comprising a computer program, which, when executed by a processor, causes the processor to perform the ground marking segmentation method of claim 13.

Images