Road sign recognition method, device, equipment and storage medium

Abstract

The present application discloses a road sign recognition method, device, equipment and storage medium, and relates to the field of data processing technology. The road sign recognition method includes: acquiring radar reflection point cloud data of a vehicle; filtering and registering the radar reflection point cloud data to obtain a point cloud data set; generating an initial image data set according to the point cloud data set, performing image repair on the initial image data set to obtain a reference sign image; performing image segmentation on the reference sign image to obtain a road sign; the method pre-processes the laser radar data in a rainstorm scene through point cloud filtering and registration, effectively improving the quality of damaged data, and then repairing the pre-processed point cloud data image to reduce the influence of noise spots generated by rain and fog in the rainstorm scene on the road sign data, and then segmenting the road sign generated by the laser radar point cloud to achieve pixel-level extraction of the road sign, so as to accurately identify the road sign in bad weather.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a road sign recognition method, device, equipment and storage medium. Background Art

[0002] Road sign detection in a parking perception environment is a key part of intelligent driving technology. The technology uses the sensor system on the vehicle to scan and analyze the surrounding environment, so that the vehicle can accurately understand information such as signs, signals and speed limit signs on the road. LiDAR is one of the commonly used sensors for road sign detection. It can efficiently obtain environmental information such as roads, obstacles and signs. It can accurately obtain the three-dimensional structural information of the surrounding environment by emitting laser beams and measuring the time it takes for them to reflect back, thereby helping the intelligent parking system to establish spatial cognition of road signs. After obtaining the three-dimensional spatial information, the road signs are detected through computer vision algorithms to realize the extraction of road signs during the intelligent parking process.

[0003] With the development of artificial intelligence in the autonomous driving industry, research on road sign detection has achieved certain breakthroughs. However, there is little research on the technology of intelligent parking road sign detection in extreme rainstorm environments. Rainstorms can cause visibility on the road to drop sharply. Factors such as rain, fog and wind can affect the performance of LiDAR, reduce the accuracy of the sensor in obtaining environmental information, and cause LiDAR and cameras to be unable to clearly capture road sign data, thus affecting the recognition accuracy of the signs.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a road sign recognition method, device, equipment and storage medium, aiming to solve the technical problem that vehicles cannot clearly capture road sign data due to bad weather, affecting the recognition accuracy of road signs.

[0006] To achieve the above objectives, the present application proposes a road sign recognition method, the road sign recognition method comprising: Obtain the radar reflection point cloud data of the vehicle; Filtering and registering the radar reflection point cloud data to obtain a point cloud data set; Generate an initial image dataset according to the point cloud dataset, and perform image restoration on the initial image dataset to obtain a reference identification image; Perform image segmentation on the reference sign image to obtain road signs.

[0007] In one embodiment, filtering and registering the radar reflection point cloud data to obtain a point cloud data set includes: Performing median filtering on the radar reflection point cloud data according to a preset filtering window to obtain filtered point cloud data; Acquire a historical point cloud data set, and perform point cloud registration based on the historical point cloud data set and the filtered point cloud data to obtain a registered point cloud data set; The registered point cloud data set is subjected to elevation filtering according to a preset elevation threshold to obtain a point cloud data set.

[0008] In one embodiment, performing median filtering on the radar reflection point cloud data according to a preset filtering window to obtain filtered point cloud data includes: Traversing the radar reflection point cloud data according to a preset filter window to obtain a reference pixel point set; Obtaining a pixel median value according to the pixel value of each pixel in the reference pixel set; The filtered point cloud data is obtained according to the pixel median value.

[0009] In one embodiment, the acquiring of the historical point cloud data set, performing point cloud registration according to the historical point cloud data set and the filtered point cloud data, and obtaining the registered point cloud data set includes: Acquire a historical point cloud data set, and map the filtered point cloud data to the historical point cloud data set to obtain reference point cloud data; Obtaining a rotation matrix and a translation matrix based on the target historical point cloud and the target to-be-registered point cloud in the reference point cloud data; The filtered point cloud data is subjected to matrix transformation according to the rotation matrix and the translation matrix to obtain a registered point cloud data set.

[0010] In one embodiment, generating an initial image dataset according to the point cloud dataset, and performing image restoration on the initial image dataset to obtain a reference identification image includes: Clustering the point cloud data set to obtain a road sign point cloud data set; Performing point cloud projection on the road sign point cloud dataset to obtain an initial image dataset; Perform image restoration on the initial image data set to obtain a reference identification image.

[0011] In one embodiment, performing image restoration on the initial image data set to obtain a reference identification image includes: Based on a preset image restoration model, the initial image data set is enhanced according to preset noise to obtain an enhanced initial image data set; Optimizing the model parameters of the preset image restoration model according to the historical image restoration errors to obtain optimized model parameters; Image restoration is performed on the initial image data set based on the optimized model parameters to obtain a reference identification image.

[0012] In one embodiment, the step of performing image segmentation on the reference sign image to obtain the road sign includes: Extracting features from the reference identification image to obtain detail features and semantic features; The reference sign image is segmented according to the detail features and the semantic features to obtain a road sign.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a road sign recognition device, the road sign recognition device comprising: Point cloud data acquisition module, used to acquire radar reflection point cloud data of the vehicle; A point cloud data processing module is used to filter and register the radar reflection point cloud data to obtain a point cloud data set; An image data restoration module is used to generate an initial image data set according to the point cloud data set, and perform image restoration on the initial image data set to obtain a reference identification image; The road sign recognition module is used to perform image segmentation on the reference sign image to obtain the road sign.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a road sign recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the road sign recognition method described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the road sign recognition method described above are implemented.

[0016] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the road sign recognition method described above are implemented.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: By filtering and pre-processing the lidar data in heavy rain scenes through point cloud filtering and point cloud registration, the quality of damaged data can be effectively improved, laying the foundation for the subsequent segmentation of road signs. The processed point cloud data is then repaired to reduce the impact of noise spots generated by rain and fog in heavy rain scenes on road sign data, thereby improving the accuracy of road sign segmentation. The road signs generated by the lidar point cloud are then segmented to achieve pixel-level extraction of road signs, which can accurately identify road signs in severe weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the road sign recognition method of the present application; Figure 2 A schematic diagram of a specific process of road sign recognition provided in Embodiment 1 of the road sign recognition method of the present application; Figure 3 A schematic diagram of a flow chart provided for the second embodiment of the road sign recognition method of the present application; Figure 4 This is a schematic diagram of the module structure of the road sign recognition device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the road sign recognition method in the embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0022] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0023] The main solution of the embodiment of the present application is: to obtain the radar reflection point cloud data of the vehicle; to filter and align the radar reflection point cloud data to obtain a point cloud data set; to generate an initial image data set based on the point cloud data set, to perform image restoration on the initial image data set to obtain a reference identification image; to perform image segmentation on the reference identification image to obtain a road sign.

[0024] In this embodiment, for the convenience of description, the following description is made by taking the road sign recognition device as the execution subject.

[0025] Since road sign detection in the existing parking perception environment is a key part of intelligent driving technology, the technology scans and analyzes the surrounding environment through the sensor system on the vehicle, so that the vehicle can accurately understand information such as signs, signals and speed limit signs on the road. LiDAR is one of the commonly used sensors for road sign detection. It can efficiently obtain environmental information such as roads, obstacles and signs. It can accurately obtain the three-dimensional structural information of the surrounding environment by emitting laser beams and measuring the time it takes for them to reflect back, thereby helping the intelligent parking system to establish spatial cognition of road signs. After obtaining the three-dimensional spatial information, the road signs are detected through computer vision algorithms to realize the extraction of road signs during the intelligent parking process.

[0026] With the development of artificial intelligence in the autonomous driving industry, research on road sign detection has achieved certain breakthroughs. However, there is little research on the technology of intelligent parking road sign detection in extreme rainstorm environments. Rainstorms can cause visibility on the road to drop sharply. Factors such as rain, fog and wind can affect the performance of LiDAR, reduce the accuracy of the sensor in obtaining environmental information, and cause LiDAR and cameras to be unable to clearly capture road sign data, thus affecting the recognition accuracy of the signs.

[0027] The present application provides a solution, which pre-processes the LiDAR data in a rainstorm scene through point cloud filtering and registration, effectively improving the quality of damaged data, and then repairing the pre-processed point cloud data image to reduce the impact of noise spots generated by rain and fog in the rainstorm scene on the road sign data, and then segmenting the road signs generated by the LiDAR point cloud to achieve pixel-level extraction of road signs, which can accurately identify road signs in severe weather.

[0028] It can be seen from the above embodiments that the present application discloses a road sign recognition method, device, equipment and storage medium, which relates to the field of data processing technology, and discloses a road sign recognition method, including: obtaining radar reflection point cloud data of a vehicle; filtering and aligning the radar reflection point cloud data to obtain a point cloud data set; generating an initial image data set based on the point cloud data set, performing image repair on the initial image data set to obtain a reference sign image; performing image segmentation on the reference sign image to obtain a road sign; the method pre-processes the lidar data in a rainstorm scene through point cloud filtering and alignment, effectively improving the quality of damaged data, and then repairing the pre-processed point cloud data image to reduce the impact of noise spots generated by rain and fog in the rainstorm scene on the road sign data, and then segmenting the road signs generated by the lidar point cloud to achieve pixel-level extraction of road signs, which can accurately identify road signs in severe weather.

[0029] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a road sign recognition device, etc. The following takes the road sign recognition device as an example to illustrate this embodiment and the following embodiments.

[0030] Based on this, the present application embodiment provides a road sign recognition method, referring to Figure 1 , Figure 1 This is a schematic flow chart of the first embodiment of the road sign recognition method of the present application.

[0031] In this embodiment, the road sign recognition method includes steps S10 to S40: Step S10, obtaining radar reflection point cloud data of the vehicle.

[0032] It should be noted that after the smart parking function is turned on, the lidar emits a laser beam. Each laser beam will be reflected when it intersects an object. The lidar receives these reflected signals and measures their time delay and intensity, thereby calculating the distance and shape information of the object, and finally integrating these data into three-dimensional point cloud data for identifying and locating spatial information in the surrounding environment.

[0033] It should be further explained that the present invention can be applied to rainstorm environments and other harsh environments that may affect radar signal reflection. The invention can be automatically started after the smart parking function is activated, or it can identify the parking environment after the smart parking function is activated, and start based on the environment to determine whether the current environment is a preset harsh environment, or it can be started according to user operation instructions after the smart parking function is activated. Among them, the method for determining the environment can be neural network classification, image recognition and other methods, which are not limited in this embodiment and can be determined according to actual conditions.

[0034] It is understandable that the radar reflection point cloud data may be point cloud data formed by a signal received by the vehicle after the radar signal emitted by the vehicle detector is reflected.

[0035] Step S20, filtering and registering the radar reflection point cloud data to obtain a point cloud data set.

[0036] It is understandable that radar reflection point cloud data is composed of radar reflection signals received by vehicle detectors in harsh environments such as heavy rain. The inevitable harsh environment will interfere with the radar signal, and the quality of the collected point cloud data will be seriously affected. Point cloud data preprocessing can be used to reduce the impact of rain and fog on point cloud scattering.

[0037] It should be noted that filtering and registration of radar reflection point cloud data are both preprocessing of radar reflection point cloud data, which can eliminate noise points caused by heavy rain and smooth the data, thereby improving the quality and accuracy of point cloud data, so that road signs in point cloud data can be more accurately identified.

[0038] It should be further explained that filtering of radar reflection point cloud data may include median filtering and elevation filtering, and registration of radar reflection point cloud data may be registration of radar reflection power data collected at the current moment and radar reflection point cloud data collected at the previous moment, and registration may be performed through an iterative closest point (ICP) algorithm.

[0039] Step S30, generating an initial image dataset according to the point cloud dataset, performing image restoration on the initial image dataset, and obtaining a reference identification image.

[0040] It should be noted that the initial image dataset is generated according to the point cloud dataset by projecting the point cloud dataset, thereby forming an image dataset including a plurality of road sign images from the point cloud dataset.

[0041] It is understandable that in severe weather conditions such as heavy rain, the images in the initial image dataset are not clear enough, which may lead to inaccurate recognition of various road signs. The image generation model GradPaint can be used to perform image restoration on the images constituting the initial image dataset to minimize the loss of point cloud set data caused by heavy rain scenes, thereby improving the quality of the sign image dataset.

[0042] It should be understood that the reference sign image may be an image including a plurality of road signs as well as surrounding environment information and interference information. The reference sign image may include road signs but may not only include road signs.

[0043] It is worth emphasizing that heavy rain will cause visibility on the road to drop sharply. Factors such as rain, fog and wind will affect the performance of lidar and reduce the accuracy of the sensor in obtaining environmental information. As a result, lidar and cameras cannot clearly capture the data of road signs, thereby affecting the recognition accuracy of the signs. This requires image generation methods to optimize the speckle noise brought to the image by rain and fog.

[0044] In a feasible implementation, step S30 may include steps A31 to A33: Step A31, clustering the point cloud dataset to obtain a road sign point cloud dataset.

[0045] It should be noted that clustering the point cloud data set may be performed using a K-means clustering algorithm to obtain a point cloud set containing road marking lines.

[0046] It should be understood that the K-means clustering algorithm is a widely used unsupervised learning method for dividing a data set into K different clusters. The basic idea of ​​this algorithm is to minimize the sum of the squared distances of all points in each cluster to the center point (i.e., centroid) of the cluster through an iterative process.

[0047] It should be understood that each cluster after K-means clustering can be understood as a road sign, and there may be multiple road signs in the road sign point cloud dataset.

[0048] Step A32, performing point cloud projection on the road sign point cloud dataset to obtain an initial image dataset.

[0049] In a specific implementation, the road sign point cloud data set is projected to obtain an initial image data set by selecting a suitable projection plane according to the application scenario, such as a front view (front projection), a side view (side projection), a top view (top view projection), etc., determining the width and height of the projection image, which will affect the resolution of the final image, and converting each point in the point cloud from a 3D spatial coordinate to a coordinate on a 2D projection plane. If the point cloud data contains reflection intensity information, the intensity value can be mapped to the pixel value of the image. If the point cloud data contains RGB color information, the color value can be directly used to generate a color image. Other attributes (such as the height of the point, the normal direction, etc.) can also be used to generate an image. During the projection process, multiple points may be mapped to the same pixel position, and the average value of the attribute values ​​of all overlapping points is calculated. According to the resolution of the projection plane, a two-dimensional array is created to store the pixel values, and the attribute value of each point (such as intensity or color) is filled into the corresponding pixel position to obtain the initial image data set.

[0050] Step A33, performing image restoration on the initial image data set to obtain a reference identification image.

[0051] It should be noted that the image restoration of the initial image data set can be performed by using the image generation model GradPaint to repair the identification image data set. The image generation model GradPaint is first trained with sample images before and after restoration, and then the initial image data set is restored by the trained image generation model GradPaint to obtain a reference identification image.

[0052] It is understandable that the reference sign image may be an image including road signs detected by the radar after restoration.

[0053] It should be noted that the image restoration of the initial image data set to obtain a reference identification image includes: enhancing the initial image data set according to preset noise based on a preset image restoration model to obtain an enhanced initial image data set; optimizing the model parameters of the preset image restoration model according to historical image restoration errors to obtain optimized model parameters; and performing image restoration on the initial image data set based on the optimized model parameters to obtain a reference identification image.

[0054] It is understandable that the pre-trained GradPaint diffusion model is used for image restoration, and gradient descent optimization is introduced in the process of generating road sign image data, so that the generated road sign image data is more coordinated with the known area of ​​the input image.

[0055] It should be noted that, based on the preset image restoration model, the initial image data set is enhanced according to the preset noise, and the enhanced initial image data set can be obtained by adding noise to the image of the initial image data set and then performing reverse denoising.

[0056] In a specific implementation, adding noise and then reverse denoising can be first adding noise to the image, and the addition of noise can refer to the following formula:

[0057] Among them, x t The image at time step t, x t-1 At time step t-1, β t The noise scheduling parameters at time step t control the intensity of the noise. I represents the identity matrix, which is used to define the covariance matrix of the noise. q(x t |x t-1 ) represents the conditional probability distribution of the forward process; N(x t , (1-β) 1 / 2 x t-1 , β t , I) is the normal distribution of the parameters.

[0058] Furthermore, after adding noise, the image can be reversely denoised by referring to the following formula:

[0059] Among them, μ θ is the denoised mean of the model prediction. 2 is a fixed noise variance.

[0060] It should be further explained that the model parameters of the preset image restoration model are optimized according to the historical image restoration error, and the optimized model parameters can be optimized by minimizing the reconstruction error. The specific optimization steps can refer to the following formula:

[0061] Among them, E q is the expectation relative to the forward process, representing the average of all possible noise paths.

[0062] In this embodiment, the radar reflection signal received in the harsh environment is preprocessed and repaired through the image generation model, which reduces the influence of noise spots generated by rain and fog in the rainstorm scene on the road sign data, thereby improving the accuracy of road sign segmentation.

[0063] The above are only feasible implementations of step S30 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S30.

[0064] Step S40: performing image segmentation on the reference sign image to obtain road signs.

[0065] It is understandable that the current road sign detection technology is still at the target detection method level and cannot effectively segment road signs at the pixel level.

[0066] It should be understood that the road sign may be a road sign that may exist in the surrounding environment identified from the image. It should be understood that there may be multiple road signs or one road sign. When there are multiple road signs in the reference sign image, image segmentation is performed based on each road sign to obtain the road sign.

[0067] In a feasible implementation, step S40 may include steps A41-A42: Step A41, extracting features from the reference identification image to obtain detail features and semantic features.

[0068] It should be noted that the feature extraction of the reference sign image can be performed by using a trained BiSegNetv2 semantic segmentation model to realize the extraction of road signs.

[0069] It should be understood that BiSegNetv2 is a neural network model for semantic segmentation tasks, which is developed based on BiSeNet (Bilateral Segmentation Network for Real-time Semantic Segmentation). BiSeNet is designed to achieve real-time semantic segmentation, especially suitable for resource-constrained environments such as embedded systems or mobile devices. The core idea of ​​BiSeNet is to process spatial details and contextual information respectively through two sub-networks, so as to achieve high-quality segmentation results while maintaining high efficiency.

[0070] It should be noted that feature extraction is performed on the reference sign image to obtain detail features and semantic features. The detail information in the road sign image can be extracted by the detail branch of the backbone network to capture detail features such as edges and textures in the image. Then, the semantic branch of the backbone network is used to extract high-level semantic information in the road sign image to help the network understand the semantic meanings of different areas in the image.

[0071] Step A42: performing image segmentation on the reference sign image according to the detail features and the semantic features to obtain a road sign.

[0072] It should be noted that the image segmentation of the reference sign image according to the detail features and the semantic features can be performed by using a bilateral guided aggregation layer to fuse the detail information and the semantic information, using the context information obtained by the semantic information extraction module to guide the output of the detail branch feature map, and generating the final road sign segmentation result through the segmentation head, thereby realizing the extraction of intelligent parking road sign segmentation.

[0073] In this embodiment, the road signs in the overhead view generated by the lidar point cloud are segmented by a semantic segmentation model to achieve pixel-level extraction of the road signs.

[0074] The above are only feasible implementations of step S40 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S40.

[0075] In the specific implementation, the specific flow chart of road sign recognition can be referred to Figure 2In the figure, point cloud data is collected when the smart parking function is started, and then the point cloud data is preprocessed by data noise reduction, data registration and data filtering. The preprocessed point cloud data is extracted to obtain a road information point cloud data set, and then the point cloud data is projected from a bird's-eye view to obtain a bird's-eye view containing road signs. The image generation model is further used to perform image restoration to obtain a restored bird's-eye view data set. The semantic segmentation model is used to segment the road signs on the bird's-eye view data set to obtain the sign segmentation result and the road signs.

[0076] This embodiment provides a road sign recognition method, which pre-processes the laser radar data by filtering and registering the point cloud in a rainstorm scene, which can effectively improve the quality of damaged data and lay the foundation for the subsequent segmentation of road signs. The processed point cloud data is then repaired, which reduces the impact of noise spots generated by rain and fog in the rainstorm scene on the road sign data and improves the accuracy of road sign segmentation. The road signs generated by the laser radar point cloud are then segmented to achieve pixel-level extraction of road signs, which can accurately identify road signs in severe weather.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S20 also includes steps S21 to S23: Step S21, performing median filtering on the radar reflection point cloud data according to a preset filtering window to obtain filtered point cloud data.

[0078] It is understandable that the radar reflection point cloud data includes a lot of point cloud data, which may include point cloud data reflected by road signs and may also include point cloud data composed of interference signals of interference objects in harsh environments.

[0079] It should be noted that the radar reflection point cloud data is subjected to median filtering according to a preset filtering window, and the filtered point cloud data is obtained by removing noise points in the point cloud data. Especially when processing radar reflection point cloud data, median filtering is to replace the value of the center point by selecting an intermediate value in a local neighborhood, thereby reducing the impact of isolated noise points.

[0080] In a feasible implementation, step S21 may include steps A211 to A213: Step A211, traverse the radar reflection point cloud data according to a preset filter window to obtain a reference pixel point set.

[0081] It should be noted that the preset filtering window may be a pre-set window of a suitable size, usually an odd number, such as 3, 5, 7, etc. This embodiment does not limit this and may be selected according to actual conditions.

[0082] It should be noted that traversing the radar reflection point cloud data according to the preset filter window may be to select a distance-based neighborhood search (such as a KD tree) or a grid-based neighborhood search. For each point, all points in its neighborhood are found, and all points in the neighborhood are used as a reference pixel point set.

[0083] It should be noted that choosing the right filter window size is very important. If the window is too small, the noise may not be removed effectively, while if the window is too large, the details may be blurred.

[0084] Step A212, obtaining a pixel median value according to the pixel value of each pixel in the reference pixel set.

[0085] In the specific implementation, the filter window n is set, the median of the pixels in each window is calculated, and the median is assigned to the point at the center of the window. The process is repeated until the entire point cloud data set is covered. The specific calculation can refer to the following formula:

[0086] Among them, Output (x, y) is the pixel value at position (x, y) in the output image, Input (x+i, y+j) is the pixel value at position (x+i, y+j) in the input image, median is the function for finding the mean, and n is the window size.

[0087] Step A213, obtaining filtered point cloud data according to the pixel point median value.

[0088] It is understandable that after traversing each point in the reference pixel set, the pixel value of each point is the median of the pixel values ​​of the original pixel value under the preset window. The pixel points of the interference signal can be eliminated, which cause abnormal pixel points in the point cloud data, thereby improving the clarity of the point cloud data and eliminating interference signals.

[0089] In this implementation, the noise points caused by heavy rain can be eliminated by median filtering, and the data can be smoothed, thereby improving the quality and accuracy of the point cloud data.

[0090] The above are only feasible implementations of step S21 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S21.

[0091] Step S22, obtaining a historical point cloud data set, performing point cloud registration based on the historical point cloud data set and the filtered point cloud data, to obtain a registered point cloud data set.

[0092] It should be understood that the filtered point cloud data can be the data obtained after filtering the point cloud data reflected by the radar at the current moment, and the historical point cloud data set can be understood as the data set after preprocessing the point cloud data reflected by the radar at the previous moment; if there is no historical data set at the previous moment, the filtered point cloud data is directly used as the registered point cloud data set.

[0093] In a feasible implementation, step S22 may include steps A221 to A223: Step A221, obtaining a historical point cloud data set, mapping the filtered point cloud data to the historical point cloud data set, and obtaining reference point cloud data.

[0094] It is understandable that the historical point cloud data set may be point cloud data of radar reflection point cloud data at the last moment after preprocessing such as filtering and registration.

[0095] It should be understood that mapping the filtered point cloud data to the historical point cloud data set to obtain reference point cloud data may be to fuse the filtered point cloud data and the historical point cloud data into the same coordinate system to facilitate subsequent alignment.

[0096] It can be understood that the reference point cloud data is a general term for all point cloud data after the filtered point cloud data and the historical point cloud data are fused into the same coordinate system. The reference point cloud data includes the point cloud data in the historical point cloud data set and the filtered point cloud data. The main purpose is to accurately align the two point cloud data sets.

[0097] Step A222, obtaining a rotation matrix and a translation matrix based on the target historical point cloud and the target to-be-registered point cloud in the reference point cloud data.

[0098] It should be noted that the rotation matrix and the translation matrix may be matrices that minimize the distance between the target historical point cloud and the target to-be-registered point cloud in the reference point cloud data.

[0099] In the specific implementation, for each point to be registered, its nearest neighbor in the reference point cloud is found to establish the correspondence between the point pairs. According to the correspondence between the point pairs, the optimal rotation matrix and translation matrix are calculated by minimizing the distance between them, so as to minimize the error between the two point clouds.

[0100] Step A223, performing matrix transformation on the filtered point cloud data according to the rotation matrix and the translation matrix to obtain a registered point cloud data set.

[0101] It is understandable that the registration of the point cloud data at the current moment and the point cloud data at the historical moment can be achieved by first synchronizing the point cloud data at the two moments into the same coordinate system.

[0102] It should be noted that the point cloud data at one of the historical moments is selected as the reference point cloud data and the point cloud data closest to the reference point cloud data is selected from the point cloud data at the current moment as the point cloud data to be registered. The reference point cloud data and the point cloud data to be registered are overlapped as much as possible, and a rotation matrix and a translation matrix are obtained that make the two point clouds overlap as much as possible.

[0103] Furthermore, the rotation matrix and the translation matrix are simultaneously considered as the registration matrices of the historical point cloud data and the filtered point cloud data. The filtered point cloud data are registered based on the registration matrix to obtain the registered point cloud data. The calculation is stopped until the convergence condition is reached. The final transformation matrix aligns the point cloud to be registered with the reference point cloud to complete the registration process.

[0104] In this implementation, preprocessing of LiDAR data in a rainstorm scene through point cloud filtering and point cloud registration can effectively improve the quality of damaged data and lay the foundation for subsequent segmentation of road signs.

[0105] The above are only feasible implementations of step S22 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S22.

[0106] Step S23, performing elevation filtering on the registered point cloud dataset according to a preset elevation threshold to obtain a point cloud dataset.

[0107] It is understandable that elevation filtering is a commonly used point cloud data processing technology, which is mainly used to remove point cloud data outside a specific height range, such as ground points or high-altitude points.

[0108] It should be noted that the point cloud data collected by the LiDAR has a variety of target objects, including buildings and trees on both sides of the road. These non-ground points will lead to a large amount of calculation for the model, resulting in unnecessary waste of computing resources. Therefore, the point cloud data is filtered by elevation filtering, the elevation distribution is observed by using the histogram distribution, and the ground points are filtered by using the threshold to obtain the driving road point cloud data.

[0109] In a specific implementation, one or more elevation thresholds are set to filter the point cloud data. For example, a minimum elevation threshold and a maximum elevation threshold can be set. The elevation value (Z coordinate) of each point is checked one by one, and those points whose elevation values ​​are within the preset threshold range are retained. The filtered points are stored in a new point cloud dataset as a point cloud dataset after elevation filtering.

[0110] This embodiment provides a road sign recognition method. It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the road sign recognition method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0111] This application also provides a road sign recognition device, please refer to Figure 4 , the road sign recognition device comprises: The point cloud data acquisition module 10 is used to acquire the radar reflection point cloud data of the vehicle; The point cloud data processing module 20 is used to filter and register the radar reflection point cloud data to obtain a point cloud data set; An image data restoration module 30 is used to generate an initial image data set according to the point cloud data set, and perform image restoration on the initial image data set to obtain a reference identification image; The road sign recognition module 40 is used to perform image segmentation on the reference sign image to obtain road signs.

[0112] The road sign recognition device provided by the present application adopts the road sign recognition method in the above embodiment, which can solve the technical problem that the vehicle cannot clearly capture the data of the road sign due to bad weather, affecting the recognition accuracy of the road sign. Compared with the prior art, the beneficial effects of the road sign recognition device provided by the present application are the same as the beneficial effects of the road sign recognition method provided by the above embodiment, and the other technical features of the road sign recognition device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0113] In one embodiment, the point cloud data processing module 20 is further used to perform median filtering on the radar reflection point cloud data according to a preset filtering window to obtain filtered point cloud data; Acquire a historical point cloud data set, and perform point cloud registration based on the historical point cloud data set and the filtered point cloud data to obtain a registered point cloud data set; The registered point cloud data set is subjected to elevation filtering according to a preset elevation threshold to obtain a point cloud data set.

[0114] In one embodiment, the point cloud data processing module 20 is further used to traverse the radar reflection point cloud data according to a preset filter window to obtain a reference pixel point set; Obtaining a pixel median value according to the pixel value of each pixel in the reference pixel set; The filtered point cloud data is obtained according to the pixel median value.

[0115] In one embodiment, the point cloud data processing module 20 is further used to obtain a historical point cloud data set, map the filtered point cloud data to the historical point cloud data set, and obtain reference point cloud data; Obtaining a rotation matrix and a translation matrix based on the target historical point cloud and the target to-be-registered point cloud in the reference point cloud data; The filtered point cloud data is subjected to matrix transformation according to the rotation matrix and the translation matrix to obtain a registered point cloud data set.

[0116] In one embodiment, the image data restoration module 30 is further used to cluster the point cloud data set to obtain a road sign point cloud data set; Performing point cloud projection on the road sign point cloud dataset to obtain an initial image dataset; Perform image restoration on the initial image data set to obtain a reference identification image.

[0117] In one embodiment, the image data restoration module 30 is further used to perform enhancement processing on the initial image data set according to preset noise based on a preset image restoration model to obtain an enhanced initial image data set; Optimizing the model parameters of the preset image restoration model according to the historical image restoration errors to obtain optimized model parameters; Image restoration is performed on the initial image data set based on the optimized model parameters to obtain a reference identification image.

[0118] In one embodiment, the road sign recognition module 40 is further used to extract features from the reference sign image to obtain detail features and semantic features; The reference sign image is segmented according to the detail features and the semantic features to obtain a road sign.

[0119] The present application provides a road sign recognition device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the road sign recognition method in the above-mentioned embodiment one.

[0120] Reference below Figure 5, which shows a schematic diagram of the structure of a road sign recognition device suitable for implementing the embodiment of the present application. The road sign recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The road sign recognition device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0121] like Figure 5 As shown, the road sign recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the road sign recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the road sign recognition device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a road sign recognition device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0122] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0123] The road sign recognition device provided by the present application adopts the road sign recognition method in the above embodiment, which can solve the technical problem that vehicles cannot clearly capture the data of road signs due to bad weather, affecting the recognition accuracy of road signs. Compared with the prior art, the beneficial effects of the road sign recognition device provided by the present application are the same as the beneficial effects of the road sign recognition method provided by the above embodiment, and the other technical features in the road sign recognition device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0124] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0125] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0126] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the road sign recognition method in the above-mentioned embodiment.

[0127] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0128] The computer-readable storage medium may be included in the road sign recognition device; or may exist independently without being assembled into the road sign recognition device.

[0129] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the road sign recognition device, the road sign recognition device: obtains the radar reflection point cloud data of the vehicle; filters and aligns the radar reflection point cloud data to obtain a point cloud data set; generates an initial image data set based on the point cloud data set, performs image restoration on the initial image data set to obtain a reference sign image; and performs image segmentation on the reference sign image to obtain a road sign.

[0130] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0133] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned road sign recognition method, and can solve the technical problem that vehicles cannot clearly capture road sign data due to bad weather, affecting the recognition accuracy of road signs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the road sign recognition method provided by the above-mentioned embodiment, and will not be repeated here.

[0134] The present application also provides a computer program product, including a computer program, which implements the steps of the road sign recognition method as described above when executed by a processor.

[0135] The computer program product provided by the present application can solve the technical problem that vehicles cannot clearly capture road sign data due to bad weather, which affects the recognition accuracy of road signs. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the road sign recognition method provided by the above embodiment, and will not be repeated here.

[0136] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A road sign recognition method, characterized in that: The road sign recognition method comprises: Obtain the radar reflection point cloud data of the vehicle; Filtering and registering the radar reflection point cloud data to obtain a point cloud data set; Generate an initial image dataset according to the point cloud dataset, and perform image restoration on the initial image dataset to obtain a reference identification image; Perform image segmentation on the reference sign image to obtain road signs.

2. The road sign recognition method according to claim 1, characterized in that: The filtering and registration processing of the radar reflection point cloud data to obtain a point cloud data set includes: Performing median filtering on the radar reflection point cloud data according to a preset filtering window to obtain filtered point cloud data; Acquire a historical point cloud data set, and perform point cloud registration based on the historical point cloud data set and the filtered point cloud data to obtain a registered point cloud data set; The registered point cloud data set is subjected to elevation filtering according to a preset elevation threshold to obtain a point cloud data set.

3. The road sign recognition method according to claim 2, characterized in that: The step of performing median filtering on the radar reflection point cloud data according to a preset filtering window to obtain filtered point cloud data includes: Traversing the radar reflection point cloud data according to a preset filter window to obtain a reference pixel point set; Obtaining a pixel median value according to the pixel value of each pixel in the reference pixel set; The filtered point cloud data is obtained according to the pixel median value.

4. The road sign recognition method according to claim 2, characterized in that: The step of acquiring a historical point cloud data set and performing point cloud registration according to the historical point cloud data set and the filtered point cloud data to obtain a registered point cloud data set includes: Acquire a historical point cloud data set, and map the filtered point cloud data to the historical point cloud data set to obtain reference point cloud data; Obtaining a rotation matrix and a translation matrix based on the target historical point cloud and the target to-be-registered point cloud in the reference point cloud data; The filtered point cloud data is subjected to matrix transformation according to the rotation matrix and the translation matrix to obtain a registered point cloud data set.

5. The road sign recognition method according to claim 1, characterized in that: The step of generating an initial image dataset according to the point cloud dataset and performing image restoration on the initial image dataset to obtain a reference identification image includes: Clustering the point cloud data set to obtain a road sign point cloud data set; Performing point cloud projection on the road sign point cloud dataset to obtain an initial image dataset; Perform image restoration on the initial image data set to obtain a reference identification image.

6. The road sign recognition method according to claim 5, characterized in that: The performing image restoration on the initial image data set to obtain a reference identification image includes: Based on a preset image restoration model, the initial image data set is enhanced according to preset noise to obtain an enhanced initial image data set; Optimizing the model parameters of the preset image restoration model according to the historical image restoration errors to obtain optimized model parameters; Image restoration is performed on the initial image data set based on the optimized model parameters to obtain a reference identification image.

7. The road sign recognition method according to claim 1, characterized in that: The step of performing image segmentation on the reference sign image to obtain the road sign includes: Extracting features from the reference identification image to obtain detail features and semantic features; The reference sign image is segmented according to the detail features and the semantic features to obtain a road sign.

8. A road sign recognition device, characterized in that: The road sign recognition device comprises: Point cloud data acquisition module, used to acquire radar reflection point cloud data of the vehicle; A point cloud data processing module is used to filter and register the radar reflection point cloud data to obtain a point cloud data set; An image data restoration module is used to generate an initial image data set according to the point cloud data set, and perform image restoration on the initial image data set to obtain a reference identification image; The road sign recognition module is used to perform image segmentation on the reference sign image to obtain the road sign.

9. A road sign recognition device, characterized in that: The device comprises: a memory, a processor, and a road sign recognition program stored in the memory and executable on the processor, wherein the road sign recognition program is configured to implement the road sign recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a road sign recognition program, and when the road sign recognition program is executed by the processor, the road sign recognition method according to any one of claims 1 to 7 is implemented.

Images