Road edge recognition method and device, computer equipment and storage medium
By fusing image and lidar data with a multimodal neural network model, the problem of poor curb recognition in existing technologies is solved, and efficient recognition of various curb lines is achieved.
Patent Information
- Application Number
- CN202510671803.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-09-19
AI Technical Summary
Existing curb recognition methods mainly rely on single-modal information, which makes it impossible to effectively identify unstructured raised curbs and non-straight curbs. The calculation is complex and the recognition effect is insufficient.
A multimodal neural network model is used to fuse image data and lidar point cloud data, and accurate identification of curbs is achieved through steps such as labeling, projection, fitting, and minimum sliding window calculation.
The accuracy and efficiency of curb recognition are improved, various types of curb lines can be recognized, and the computational complexity is reduced.
Smart Images
Figure CN120673357A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of road traffic technology, and in particular, to a curb recognition method, device, computer equipment, and storage medium. Background Art
[0002] Road boundaries are a crucial element in assisted and autonomous driving systems. This information can be obtained through high-precision maps or real-time road boundary perception technology. Real-time road boundary perception typically requires curb recognition technology.
[0003] Traditional curb recognition methods use LiDAR point clouds to extract structured protrusions on curbs, or use camera images to identify 2D curb lines. Both methods rely solely on single-modal information—data from a single LiDAR or camera. Each has information limitations, resulting in insufficient recognition. For example, relying solely on LiDAR point clouds is essentially incapable of identifying unstructured protrusions (such as gentle slopes and low curbs), while relying solely on cameras is not conducive to identifying non-linear curbs and typically requires projection transformation into 3D space, which is computationally complex. Summary of the Invention
[0004] The embodiments of the present application provide a curb recognition method, apparatus, computer device, and storage medium.
[0005] A first aspect of an embodiment of the present application provides a curb recognition method, comprising:
[0006] Acquire image data and lidar point cloud data of different types of roadside scenes, and project the lidar point cloud data onto the image data for verification and alignment;
[0007] Annotating the roadside data in the aligned lidar point cloud data to obtain a roadside true value point set;
[0008] Constructing a multimodal neural network model, and inputting the roadside true value point set into the multimodal neural network model for model training;
[0009] Inputting the laser radar point cloud data of the target road scene into the trained multimodal neural network model to obtain a roadside discrete point set;
[0010] Performing interval elevation plane fitting on the roadside discrete point set along the road direction, and projecting the roadside discrete points within the interval onto the elevation fitting plane;
[0011] A minimum sliding window calculation is performed on the projected roadside discrete point set, the roadside discrete point set in each window is sampled to obtain a roadside sampling point set, and the roadside sampling point set is projected as image data.
[0012] In an optional embodiment of the present application, the process of labeling the roadside data in the laser radar point cloud data to obtain a roadside true value point set includes:
[0013] Set the annotation perspective;
[0014] Based on the marked perspective, curb points are marked in the multi-frame curb data of each curb road scene to obtain a curb true value point set. The curb points are points on the curb side facade or on both sides of the road boundary whose width is less than a preset value.
[0015] In an optional embodiment of the present application, the process of constructing a multimodal neural network model and inputting the roadside true value point set into the multimodal neural network model for model training includes:
[0016] Build a multimodal neural network model;
[0017] The image data features and the lidar point cloud data features are used as input, the roadside true value point set is used as true value supervision, the roadside point semantic segmentation is performed, and the roadside points are output.
[0018] In an optional embodiment of the present application, the process of fitting the discrete roadside points along the road direction to an interval elevation plane and projecting the discrete roadside points in the interval onto the elevation fitting plane includes:
[0019] Divide the roadside discrete point set into N intervals along the vehicle's travel direction;
[0020] Perform elevation plane fitting on the discrete points along the road in each interval to obtain an elevation fitting plane;
[0021] Project the discrete points along the road in each interval onto the elevation fitting plane.
[0022] In an optional embodiment of the present application, the process of performing minimum sliding window calculation on the projected roadside discrete point set includes:
[0023] Set the length and width of the window;
[0024] Starting from one end of the projected roadside discrete point set, a 2D horizontal sliding window is made according to the set window length and width;
[0025] Calculate the minimum number of windows that contain all discrete roadside points that satisfy x∈[xi,xi+step), and record the window position and all discrete roadside points belonging to the window, where x is the horizontal coordinate of the discrete roadside point, xi is the starting x-coordinate of the i-th interval, and step is the window length;
[0026] Repeatedly calculate the minimum number of windows along the y direction, and then along the x direction until reaching the other end of the roadside discrete point set. The x direction is the vehicle driving direction, and the y direction is the direction perpendicular to the x direction.
[0027] In an optional embodiment of the present application, the process of sampling the roadside discrete point set in each window to obtain the roadside sampling point set includes:
[0028] The number of sampling points in each window is determined according to the set window length and width;
[0029] Randomly sample the discrete point set of the roadside in each window according to the number of sampling points in each window;
[0030] The sampling points randomly sampled from all windows are combined to form a roadside sampling point set.
[0031] In an optional embodiment of the present application, before obtaining image data and lidar point cloud data of various types of roadside scenes, the method further includes: performing external parameter calibration between the camera and the lidar, and between the lidar and the vehicle-mounted positioning sensor.
[0032] In an optional embodiment of the present application, the process of projecting and verifying the laser radar point cloud data onto the image data includes: projecting the laser radar point cloud data onto the exposure image corresponding to the 0ms time according to the external parameters of the laser radar to the camera.
[0033] In an optional embodiment of the present application, before acquiring image data and lidar point cloud data of various types of roadside scenes, the method further includes:
[0034] The 0ms moment of each 100ms cycle is used as the time alignment benchmark to calculate the vehicle position and angle changes within each cycle;
[0035] The point-by-point timestamp assigned by the LiDAR to the LiDAR point cloud data is interpolated to obtain the vehicle motion situation point by point to 0ms;
[0036] Compensation and restoration are performed based on the vehicle position change, vehicle angle change, and vehicle motion to obtain the laser radar point cloud data after vehicle motion compensation.
[0037] In an optional embodiment of the present application, after labeling the roadside data in the lidar point cloud data to obtain a roadside true value point set, the method further includes: cleaning the roadside true value point set.
[0038] In an optional embodiment of the present application, after constructing a multimodal neural network model and inputting the roadside true value point set into the multimodal neural network model for model training, the method further includes: screening the multimodal neural network model after a preset number of training rounds to obtain an optimized model.
[0039] A second aspect of an embodiment of the present application provides a curb recognition device, comprising:
[0040] An acquisition and verification module is used to acquire image data and lidar point cloud data of different types of roadside scenes, and project the lidar point cloud data onto the image data for verification and alignment;
[0041] A data annotation module is used to annotate the roadside data in the aligned lidar point cloud data to obtain a roadside true value point set;
[0042] A model training module is used to construct a multimodal neural network model and input the roadside true value point set into the multimodal neural network model for model training;
[0043] A roadside discrete point set acquisition module is used to input the laser radar point cloud data of the target road scene into the trained multimodal neural network model to obtain a roadside discrete point set;
[0044] An interval fitting module is used to fit the discrete roadside points along the road direction to an interval elevation plane, and project the discrete roadside points within the interval onto the elevation fitting plane;
[0045] The data sampling module is used to perform minimum sliding window calculation on the projected roadside discrete point set, sample the roadside discrete point set in each window to obtain a roadside sampling point set, and project the roadside sampling point set into image data.
[0046] According to a third aspect of an embodiment of the present application, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.
[0047] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.
[0048] This application obtains and fuses multimodal information of image data of various types of roadside scenes and lidar point cloud data, adopts a multimodal neural network model for learning, and performs post-processing such as interval elevation plane fitting and minimum sliding window calculation to obtain roadside sampling points, and ultimately realize the identification of various types of roadside lines. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0050] Figure 1 A flowchart of a curb recognition method provided in one embodiment of the present application;
[0051] Figure 2 The figure is a schematic diagram of projecting a set of discrete roadside points obtained in a roadside scene onto an elevation fitting plane;
[0052] Figure 3 The figure is a schematic diagram of projecting the discrete point set of the roadside obtained in another roadside scenario onto the elevation fitting plane;
[0053] Figure 4 For Figure 2 Schematic diagram of the minimum sliding window calculation along the discrete point set;
[0054] Figure 5 For Figure 3 Schematic diagram of the minimum sliding window calculation along the discrete point set;
[0055] Figure 6 For Figure 4 A schematic diagram of obtaining a roadside sampling point set by sampling a roadside discrete point set in each window;
[0056] Figure 7 For Figure 5 A schematic diagram of obtaining a roadside sampling point set by sampling a roadside discrete point set in each window;
[0057] Figure 8 A schematic diagram of the structure of a curb recognition device provided in one embodiment of the present application;
[0058] Figure 9 A schematic diagram of the computer device structure provided for one embodiment of the present application. DETAILED DESCRIPTION
[0059] In the process of realizing the present application, the inventors discovered that the current curb recognition method has a relatively simple recognition mode and cannot realize the recognition of various types of curb routes.
[0060] In order to solve the above problems, the present invention provides a roadside recognition method, such as Figure 1 As shown, including:
[0061] S101. Obtain image data and lidar point cloud data of different types of roadside scenes, and project the lidar point cloud data onto the image data for verification and alignment.
[0062] Specifically, this embodiment requires acquiring image data and LiDAR point cloud data for various types of roadside scenes. Image data can be acquired using a camera, while LiDAR point cloud data can be acquired using a LiDAR. The collected roadside scenes should ideally be those with a wide field of view and unobstructed roads to improve recognition accuracy. If geographically restricted, SLAM technology can be used to assist in acquiring accurate vehicle position and posture, thereby assisting in the acquisition of image data and LiDAR point cloud data for various types of roadside scenes.
[0063] After acquiring image data and LiDAR point cloud data for various roadside scenarios, projection verification of the LiDAR point cloud data onto the image data is required to ensure multimodal data alignment. Specifically, based on the LiDAR-to-camera extrinsic parameters, the LiDAR point cloud data can be projected onto the corresponding 0ms exposure image to verify multimodal data alignment.
[0064] S102: Annotate the roadside data in the aligned lidar point cloud data to obtain a roadside true value point set.
[0065] In S101, the image data and the lidar point cloud data have been aligned for multimodal data. Therefore, in this embodiment, it is only necessary to mark the curb data related to the curb in the lidar point cloud data.
[0066] Specifically, in this embodiment, a labeling perspective is first set. Typically, for ease of labeling, a bird's-eye view is selected as the labeling perspective. Based on the labeling perspective, curb points are annotated in multiple frames of curb data for each curb scene to obtain a set of true curb points. In this embodiment, curb points are defined as all points on the curb's side elevation or the actual road boundary whose width is less than a preset value, thereby distinguishing all curb points. This preset value can be set to 5 cm for conventional road scenes. To cover a variety of curb types, this embodiment labels at least 20 frames for each curb scene. Furthermore, to avoid high similarity between consecutive frames, frame extraction is performed for the same curb scene. Furthermore, after collecting as many different curb scenes as possible, the cumulative number of true curb points after labeling is no less than 5,000. Of course, this embodiment only provides a set of labeled frame values; the specific value can be adjusted based on the complexity of the actual labeled curb scene, and this embodiment does not impose any specific restrictions.
[0067] S103: Construct a multimodal neural network model, and input the roadside true value point set into the multimodal neural network model for model training.
[0068] S104: Input the laser radar point cloud data of the target road scene into the trained multimodal neural network model to obtain a roadside discrete point set.
[0069] After obtaining the roadside truth point set in S102, it is necessary to select an appropriate multimodal neural network model to perform network learning training on the roadside truth point set. Among them, the multimodal neural network model is selected according to the actual situation. In this embodiment, BEVFusion is used as an example to construct a multimodal semantic segmentation task. BEVFusion is a multi-sensor fusion technology, which is mainly used to fuse laser radar (LiDAR) and camera data to generate a unified bird's-eye view (BEV) representation, thereby improving the accuracy and robustness of 3D object detection and segmentation. With image data features and laser radar point cloud data features as input, feature fusion is completed in 3D voxel space. With the roadside truth point set as the true value supervision, the semantic segmentation of the roadside points is completed, and the roadside points are inferred and output to form a discrete roadside point set. This embodiment uses a multimodal neural network model to integrate image data and laser radar point cloud data from different modalities. Each modality can provide unique information for specific tasks, thereby enhancing the model's understanding and reasoning capabilities and improving the integrity and accuracy of the information.
[0070] S105 , performing interval elevation plane fitting on the roadside discrete point set along the road direction, and projecting the roadside discrete points within the interval onto the elevation fitting plane.
[0071] After obtaining the roadside discrete point set through S103 and S104, the roadside discrete point set is input into the interval elevation plane fitting module for interval elevation plane fitting. In this process, the vehicle is divided into N intervals along the direction of travel (x-axis direction). By performing RANSAC (Random Sample Consensus) elevation plane fitting on the roadside discrete points in each interval, the elevation fitting plane is obtained, and all the roadside discrete points in the interval are projected onto the fitting plane, thereby eliminating the influence of the elevation difference in the same interval. At the same time, the possibility of processing the road slope can be retained to a certain extent through N intervals. Figure 2 and Figure 3 Schematic diagrams of projecting the discrete roadside point sets obtained under two different roadside scenarios onto the elevation fitting plane.
[0072] S106 , performing minimum sliding window calculation on the projected roadside discrete point set, sampling the roadside discrete point set in each window to obtain a roadside sampling point set, and projecting the roadside sampling point set into image data.
[0073] Specifically, the minimum sliding window calculation can reduce time complexity and space complexity, and has strong flexibility. First, set the length and width of the window. Then, starting from one end (x_min) of the projected roadside discrete point set, make a 2D horizontal (i.e., y-direction) sliding window according to the set window length and width, calculate the minimum number of windows containing all roadside discrete points that satisfy x∈[xi,xi+step), and record the window position and all roadside discrete points belonging to the window, where x is the horizontal coordinate of the roadside discrete point, xi is the starting x-coordinate of the i-th interval, and step is the window length. Finally, repeat the calculation of the minimum number of windows along the y direction, and then along the x direction until reaching the other end (x_max) of the roadside discrete point set. As shown Figure 4 Shown is the Figure 2 The schematic diagram of the minimum sliding window calculation along the discrete point set is as follows: Figure 5 Shown is the Figure 3 Schematic diagram of the minimum sliding window calculation along a discrete point set.
[0074] After the minimum sliding window calculation is completed, the roadside discrete point set in each window needs to be sampled. Among them, the number of sampling points and the sampling method can be determined according to the set window length and width. According to the number of sampling points in each window, the roadside discrete point set in each window is randomly sampled. The sampling points randomly sampled from all windows are combined to form a roadside sampling point set. For example, a 0.5m*0.5m size window is used in the x and y directions, and only one random sampling point can be retained. The sampling points of all windows form a roadside sampling point set, and a roadside sampling point set with a sampling interval of about 0.5m can be obtained, which is sufficient to discretize and describe various complex line types. For example Figure 6 Shown is the Figure 4 The schematic diagram of sampling the roadside discrete point set in each window to obtain the roadside sampling point set is as follows: Figure 7 Shown is the Figure 5 Schematic diagram of sampling the roadside discrete point set within each window in to obtain the roadside sampling point set. After obtaining the roadside sampling point set, the roadside sampling point set can be projected into image data for verification.
[0075] This embodiment obtains and fuses multimodal information of image data of various types of roadside scenes and lidar point cloud data, adopts a multimodal neural network model for learning, and performs post-processing such as interval elevation plane fitting and minimum sliding window calculation to obtain roadside sampling points, thereby ultimately realizing the recognition of various types of roadside lines.
[0076] In some optional embodiments, before acquiring image data and lidar point cloud data of various types of roadside scenes, the roadside recognition method further includes: performing external parameter calibration between the camera and the lidar, and between the lidar and the vehicle-mounted positioning sensor.
[0077] Specifically, during the initial preparation phase, the vehicle-mounted data acquisition system's LiDAR and camera devices undergo time synchronization and external parameter calibration. Conventional calibration methods are first used to calibrate external parameters between the camera and LiDAR, and between the LiDAR and the vehicle-mounted positioning sensor (the origin of the vehicle's coordinate system). Calibration alignment is verified using static targets with the vehicle stationary, ensuring that translation and rotation parameter calibrations meet accuracy requirements. For example, translation parameter calibration meets 3cm accuracy requirements, and rotation parameter calibration meets 0.2° accuracy requirements.
[0078] Then adjust the camera triggering time to align the exposure time with the start time of the LiDAR. For example, if the data frequency is 10Hz and the period is 100ms, the exposure time of 0ms in each period corresponds to the start time of the LiDAR scan.
[0079] Finally, the alignment was verified by using a static target while the vehicle was in motion, confirming that the time synchronization error was less than 10ms.
[0080] In some optional embodiments, before acquiring image data and lidar point cloud data of various types of roadside scenes, the roadside recognition method further includes: performing vehicle motion compensation on the lidar point cloud data based on spatiotemporal synchronization to align it with the image exposure time.
[0081] Specifically, the 0ms moment of each 100ms cycle is used as the time alignment reference to calculate the vehicle's position and angle change within each cycle. This position and angle change can be calculated using a GNSS+IMU or SLAM algorithm. The point-by-point timestamp assigned by the LiDAR to the LiDAR point cloud data is interpolated to obtain the vehicle's motion up to the 0ms point. Compensation is performed based on the vehicle's position change, angle change, and motion to obtain the compensated LiDAR point cloud data.
[0082] In some optional embodiments, after labeling the roadside data in the lidar point cloud data to obtain a roadside true value point set, the roadside recognition method further includes: cleaning the roadside true value point set.
[0083] Specifically, in this embodiment, the process of annotating the curb data in the LiDAR point cloud data primarily involves manual annotation. However, during manual annotation, different people may have different understandings of the annotation standards. Therefore, after the annotation of the ground-truth curb point set is completed, the annotated data can be cleaned based on the annotation quality to ensure that the distribution and number of ground-truth curb points in similar road scenes are reasonable and uniform, without significant differences.
[0084] In some optional embodiments, after constructing a multimodal neural network model and inputting the roadside truth point set into the multimodal neural network model for model training, the roadside recognition method further includes: screening the multimodal neural network model after a preset number of training rounds to obtain an optimized model.
[0085] Specifically, since the convergence direction of the multimodal neural network model has a certain degree of randomness during the supervised learning training process. At the same time, the subsequent post-processing processes such as interval elevation plane fitting are more sensitive to false detections, insensitive to missed detections and can provide a safety net. Therefore, this embodiment can manually screen out multimodal neural network models with under-segmentation tendencies as optimization models to reduce the frequency of false detections. The specific manual screening method can select a model with a relatively high Precision index, or a model with a concave and narrowing phenomenon along the discrete point set in the visualization.
[0086] The solutions in the embodiments of the present application can be implemented using various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.
[0087] In order to make the technical solutions and advantages of the embodiments of the present application more clearly understood, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, and are not an exhaustive list of all the embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other unless they conflict.
[0088] It should be understood that, although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0089] Example 2
[0090] like Figure 8 As shown, one embodiment of the present application provides a curb recognition device, comprising:
[0091] An acquisition and verification module is used to acquire image data and lidar point cloud data of different types of roadside scenes, and project the lidar point cloud data onto the image data for verification and alignment;
[0092] A data annotation module is used to annotate the roadside data in the aligned lidar point cloud data to obtain a roadside true value point set;
[0093] A model training module is used to construct a multimodal neural network model and input the roadside true value point set into the multimodal neural network model for model training;
[0094] A roadside discrete point set acquisition module is used to input the laser radar point cloud data of the target road scene into the trained multimodal neural network model to obtain a roadside discrete point set;
[0095] An interval fitting module is used to fit the discrete roadside points along the road direction to an interval elevation plane, and project the discrete roadside points within the interval onto the elevation fitting plane;
[0096] The data sampling module is used to perform minimum sliding window calculation on the projected roadside discrete point set, sample the roadside discrete point set in each window to obtain a roadside sampling point set, and project the roadside sampling point set into image data.
[0097] The specific limitations of the aforementioned curb recognition device can be found in the limitations of the curb recognition method described above and will not be further elaborated here. Each module in the aforementioned curb recognition device may be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules may be embedded in or independent of a processor within a computer device in hardware form, or may be stored in a computer device memory in software form, allowing the processor to call and execute the corresponding operations of each module.
[0098] Example 3
[0099] In one embodiment, a computer device is provided. The internal structure diagram of the computer device can be as follows: Figure 9As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for curb recognition as described above is implemented. It includes: a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any step in the method for curb recognition as described above is implemented.
[0100] Example 4
[0101] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any step in the upper curb recognition method can be implemented.
[0102] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0103] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0104] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.
[0106] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.
[0107] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.
Claims
1. A curb recognition method, characterized in that: include: Acquire image data and lidar point cloud data of different types of roadside scenes, and project the lidar point cloud data onto the image data for verification and alignment; Annotating the roadside data in the aligned lidar point cloud data to obtain a roadside true value point set; Constructing a multimodal neural network model, and inputting the roadside true value point set into the multimodal neural network model for model training; Inputting the laser radar point cloud data of the target road scene into the trained multimodal neural network model to obtain a roadside discrete point set; Performing interval elevation plane fitting on the roadside discrete point set along the road direction, and projecting the roadside discrete points within the interval onto the elevation fitting plane; A minimum sliding window calculation is performed on the projected roadside discrete point set, the roadside discrete point set in each window is sampled to obtain a roadside sampling point set, and the roadside sampling point set is projected as image data.
2. The method according to claim 1, characterized in that The process of labeling the roadside data in the aligned laser radar point cloud data to obtain a roadside true value point set includes: Set the annotation perspective; Based on the marked perspective, curb points are marked in the multi-frame curb data of each curb road scene to obtain a curb true value point set. The curb points are points on the curb side facade or on both sides of the road boundary whose width is less than a preset value.
3. The method according to claim 1, characterized in that The process of constructing a multimodal neural network model and inputting the roadside true value point set into the multimodal neural network model for model training includes: Build a multimodal neural network model; The image data features and the lidar point cloud data features are used as input, the roadside true value point set is used as true value supervision, the roadside point semantic segmentation is performed, and the roadside points are output.
4. The method according to claim 1, wherein The process of fitting the discrete roadside points along the road direction to an interval elevation plane and projecting the discrete roadside points in the interval onto the elevation fitting plane includes: Divide the roadside discrete point set into N intervals along the vehicle's travel direction; Perform elevation plane fitting on the discrete points along the road in each interval to obtain an elevation fitting plane; Project the discrete points along the road in each interval onto the elevation fitting plane.
5. The method according to claim 1, wherein The process of performing minimum sliding window calculation on the projected roadside discrete point set includes: Set the length and width of the window; Starting from one end of the projected roadside discrete point set, a 2D horizontal sliding window is made according to the set window length and width; Calculate the minimum number of windows that contain all discrete roadside points that satisfy x∈[xi,xi+step), and record the window position and all discrete roadside points belonging to the window, where x is the horizontal coordinate of the discrete roadside point, xi is the starting x-coordinate of the i-th interval, and step is the window length; Repeatedly calculate the minimum number of windows along the y direction, and then along the x direction until reaching the other end of the roadside discrete point set. The x direction is the vehicle driving direction, and the y direction is the direction perpendicular to the x direction.
6. The method according to claim 5, characterized in that The process of sampling the roadside discrete point set in each window to obtain the roadside sampling point set includes: The number of sampling points in each window is determined according to the set window length and width; Randomly sample the discrete point set of the roadside in each window according to the number of sampling points in each window; The sampling points randomly sampled from all windows are combined to form a roadside sampling point set.
7. The method according to claim 1, characterized in that Before acquiring image data and lidar point cloud data of various types of roadside scenes, the method further includes: performing external parameter calibration between the camera and the lidar, and between the lidar and the vehicle-mounted positioning sensor.
8. The method according to claim 1, characterized in that The process of projecting the laser radar point cloud data onto the image data for verification and alignment includes: projecting the laser radar point cloud data onto the exposure image corresponding to the 0ms time according to the external parameters of the laser radar to the camera.
9. The method according to claim 1, characterized in that Before acquiring image data and lidar point cloud data of different types of roadside scenes, the method further includes: The 0ms moment of each 100ms cycle is used as the time alignment benchmark to calculate the vehicle position and angle changes within each cycle; The point-by-point timestamp assigned by the LiDAR to the LiDAR point cloud data is interpolated to obtain the vehicle motion situation point by point to 0ms; Compensation and restoration are performed based on the vehicle position change, vehicle angle change, and vehicle motion to obtain the laser radar point cloud data after vehicle motion compensation.
10. The method according to claim 1, characterized in that After labeling the roadside data in the laser radar point cloud data to obtain a roadside true value point set, the method further includes: cleaning the roadside true value point set.
11. The method according to claim 1, wherein After constructing a multimodal neural network model and inputting the roadside true value point set into the multimodal neural network model for model training, the method further includes: screening the multimodal neural network model after a preset number of training rounds to obtain an optimized model.
12. A curb recognition device, characterized in that: include: An acquisition and verification module is used to acquire image data and lidar point cloud data of different types of roadside scenes, and project the lidar point cloud data onto the image data for verification and alignment; A data annotation module is used to annotate the roadside data in the aligned lidar point cloud data to obtain a roadside true value point set; A model training module is used to construct a multimodal neural network model and input the roadside true value point set into the multimodal neural network model for model training; A roadside discrete point set acquisition module is used to input the laser radar point cloud data of the target road scene into the trained multimodal neural network model to obtain a roadside discrete point set; An interval fitting module is used to fit the discrete roadside points along the road direction to an interval elevation plane, and project the discrete roadside points within the interval onto the elevation fitting plane; The data sampling module is used to perform minimum sliding window calculation on the projected roadside discrete point set, sample the roadside discrete point set in each window to obtain a roadside sampling point set, and project the roadside sampling point set into image data.
13. A computer device comprising: The method comprises a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.