Road surface traffic sign extraction method and device, electronic equipment and storage medium

Abstract

The application provides a road traffic sign extraction method and device, electronic equipment and storage medium, comprising: acquiring point cloud data of a target scene, and determining a target projection image corresponding to the target scene based on the point cloud data; wherein the content displayed by the target projection image at least includes a road traffic sign; through a pre-trained target detection model, target detection processing is performed on the target projection image to obtain sign attribute data corresponding to the road traffic sign; wherein the target detection model is obtained by distillation training on a lightweight detection model; based on the sign attribute data, a target local image and a target template image corresponding to the road traffic sign are determined, and a sign boundary corresponding to the road traffic sign is extracted from the target local image according to the target template image. The application can effectively reduce the requirement for hardware equipment, and can also better balance the efficiency and effect of extracting the road traffic sign.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a road traffic sign extraction method and device, electronic equipment and storage medium. BACKGROUND

[0002] Current traffic sign extraction methods mainly have two categories: (1) a ground traffic sign detection and recognition algorithm based on deep learning, but in order to obtain a high-precision recognition result, a relatively complex model needs to be designed, which not only has a high requirement for hardware devices, but also has a slow running speed; (2) a ground traffic sign detection and recognition algorithm based on structure matching, the recognition result of which depends on traffic sign feature clarity and data processing results, which not only increases the time length consumed by preprocessing, but also results in an unsatisfactory recognition result due to complex actual road conditions (such as wear and tear, obstruction, etc.). SUMMARY

[0003] Therefore, the present application aims to provide a road traffic sign extraction method and device, electronic equipment and storage medium, which can effectively reduce the requirement for hardware devices and also better balance the efficiency and effect of extracting road traffic signs.

[0004] In a first aspect, an embodiment of the present application provides a road traffic sign extraction method, comprising: acquiring point cloud data of a target scene, and determining a target projection image corresponding to the target scene based on the point cloud data; wherein the target projection image displays content including at least road traffic signs; performing target detection processing on the target projection image through a pre-trained target detection model to obtain sign attribute data corresponding to the road traffic signs; wherein the target detection model is obtained by distillation training a lightweight detection model; determining a target local image and a target template image corresponding to the road traffic signs based on the sign attribute data, and extracting a sign boundary corresponding to the road traffic signs from the target local image according to the target template image.

[0005] In an embodiment, the method further comprises: acquiring a training data set; training a pre-constructed initial detection model using the training data set, and distillation training a pre-constructed lightweight detection model using the trained initial detection model; performing compression processing on the distillation-trained lightweight detection model, and converting the format of the compressed lightweight detection model to a specified format to obtain a target detection model; wherein the specified format includes an ONNX format.

[0006] In an embodiment, the sign attribute data comprises a sign type and a sign bounding box; and based on the sign attribute data, determining a target local image and a target template image corresponding to the road traffic sign comprises: determining a national standard size corresponding to the road traffic sign according to a bounding box size of the sign bounding box, and determining a target template image corresponding to the road traffic sign based on the national standard size corresponding to the road traffic sign and the sign type; and determining a target local image corresponding to the road traffic sign based on a preset pixel number of outer expansion, the sign bounding box, the target projection image and the target template image.

[0007] In an embodiment, determining a target local image corresponding to the road traffic sign based on a preset pixel number of outer expansion, the sign bounding box, the target projection image and the target template image comprises: performing cropping on the target projection image based on the preset pixel number of outer expansion and the sign bounding box to obtain an initial local image corresponding to the road traffic sign; determining a gray histogram of the initial local image; sequentially accumulating pixel numbers corresponding to gray values in the gray histogram in a descending order of the gray values until a pixel ratio obtained by the accumulation is greater than a non-zero pixel ratio of the target template image, and determining a target gray value; determining a segmentation threshold value corresponding to the initial local image by using the Otsu algorithm; if the target gray value is greater than the segmentation threshold value, determining the target gray value as a binarization threshold value; or if the segmentation threshold value is greater than the target gray value, determining the segmentation threshold value as the binarization threshold value; and performing binarization processing on the initial local image by using the binarization threshold value to obtain a target local image corresponding to the road traffic sign.

[0008] In an embodiment, the sign attribute data further comprises a sign angle; and extracting a sign boundary corresponding to the road traffic sign from the target local image according to the target template image comprises: cutting a preset angle threshold range by using a preset step size to obtain a plurality of angle threshold values; superimposing the sign angle corresponding to the road traffic sign with each of the angle threshold values to obtain a plurality of candidate rotation angles; for each of the candidate rotation angles, rotating the target local image according to the candidate rotation angle, determining a difference degree between the target local image after rotation and the target template image, and determining a sign position under the candidate rotation angle based on the difference degree; determining a target rotation angle from each of the candidate rotation angles according to the difference degree corresponding to each of the candidate rotation angles, and determining the sign position under the target rotation angle as a target sign position; and extracting a sign boundary corresponding to the road traffic sign from the target local image based on the target rotation angle and the target sign position.

[0009] In an embodiment, the difference degree between the rotated target local image and the target template image is determined, and the mark position under the candidate rotation angle is determined based on the difference degree, including: adjusting the size of the target template image to obtain an adjusted template image according to the size of the outer frame of the mark; determining a first difference degree between the rotated target local image and the target template image and a first mark position of the road traffic sign in the target local image through pixel-by-pixel translation matching; and determining a second difference degree between the rotated target local image and the adjusted template image and a second mark position of the road traffic sign in the target local image through pixel-by-pixel translation matching; if the first difference degree is less than the second difference degree, the first mark position is determined as the mark position under the candidate rotation angle; or if the first difference degree is greater than the second difference degree, the second mark position is determined as the mark position under the candidate rotation angle.

[0010] In an embodiment, the target projection image corresponding to the target scene is determined based on the point cloud data, including: projecting each point in the point cloud data to a specified plane based on the coordinate data of each point, and determining the gray value of each pixel point in the specified plane based on the reflection intensity data of each point to obtain an initial projection image; and performing slice processing on the initial projection image to obtain the target projection image corresponding to the target scene.

[0011] In a second aspect, an embodiment of the present application further provides a road traffic sign extraction device, including: a projection image determination module configured to obtain point cloud data of a target scene, and determine a target projection image corresponding to the target scene based on the point cloud data; wherein the target projection image displays at least a road traffic sign; a mark detection module configured to perform target detection processing on the target projection image through a pre-trained target detection model to obtain mark attribute data corresponding to the road traffic sign; wherein the target detection model is obtained by distillation training of a lightweight detection model; and a mark extraction module configured to determine a target local image and a target template image corresponding to the road traffic sign based on the mark attribute data, and extract a mark boundary corresponding to the road traffic sign from the target local image according to the target template image.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device including a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of the first aspect.

[0013] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method according to any one of the first aspect.

[0014] The method and device for extracting road traffic signs, the electronic device and the storage medium provided by the embodiments of the present application first acquire point cloud data of a target scene, determine a target projection image corresponding to the target scene based on the point cloud data, and the content displayed by the target projection image at least includes road traffic signs. Then, a target detection model obtained by distillation training of a lightweight detection model is used to perform target detection processing on the target projection image to obtain sign attribute data corresponding to the road traffic signs. Then, based on the sign attribute data, a target local image and a target template image corresponding to the road traffic signs are determined, and then the target template image is used to extract a sign boundary corresponding to the road traffic signs from the target local image. The above method first uses point cloud data to determine a target projection image corresponding to a target scene, and then combines the advantages of deep learning algorithms and image post-processing algorithms, uses a lightweight and simplified target detection model to perform target detection processing on the target projection image to obtain corresponding sign attribute data, which can not only reduce the requirements on hardware devices, but also effectively improve the machine recognition efficiency. At the same time, the image post-processing algorithm is used to further optimize the machine recognition effect, and based on the sign attribute data, a corresponding target local image and a target template image are determined, and then the target template image is used to extract a sign boundary corresponding to the road traffic signs from the target local image, which not only ensures the reliability of the recognition effect, but also meets the real-time requirement. In addition, the embodiments of the present application also have good recognition effect on worn and blocked road traffic signs, and reduce the strong dependence of traditional structure matching algorithms on sign features.

[0015] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present application more apparent, the following preferred embodiments are specifically described with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0018] Figure 1 A flowchart of a road traffic sign extraction method provided by an embodiment of the present application is shown in the figure.

[0019] Figure 2 A flowchart of another road traffic sign extraction method provided by an embodiment of the present application is shown in the figure.

[0020] Figure 3 An extraction effect diagram of a road traffic sign in a large scene provided by an embodiment of the present application is shown in the figure.

[0021] Figure 4 A detailed diagram of the extraction effect of a road traffic sign provided by an embodiment of the present application is shown in the figure.

[0022] Figure 5 An extraction effect diagram of a road traffic sign in a wear and loss scene provided by an embodiment of the present application is shown in the figure.

[0023] Figure 6 A structural diagram of a road traffic sign extraction device provided by an embodiment of the present application is shown in the figure.

[0024] Figure 7 A structural diagram of an electronic device provided by an embodiment of the present application is shown in the figure. Specific embodiments

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be described clearly and completely in combination with embodiments. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0026] Current traffic sign extraction methods mainly have two categories: (1) ground traffic sign detection and recognition algorithm based on deep learning, which mainly includes two tasks of target positioning and recognition, has higher requirements for hardware devices, and slower running speed, which is not conducive to the popularization and landing of the algorithm; (2) ground traffic sign detection and recognition algorithm based on structure matching, which mainly includes data preprocessing, image segmentation technology, feature extraction, feature matching and other steps, wherein the time consumed by data preprocessing is longer, and the recognition effect for the actual road environment is not ideal.

[0027] Based on this, the present application provides a road traffic sign extraction method, device, electronic equipment and storage medium, which can effectively reduce the requirements for hardware devices, and also better balance the efficiency and effect of extracting road traffic signs.

[0028] In order to facilitate the understanding of the present embodiment, first of all, a road traffic sign extraction method disclosed by the present embodiment is introduced in detail, referring to the flow diagram of the road traffic sign extraction method shown in Figure 1 The method mainly includes the following steps S102 to S106:

[0029] Step S102, acquiring point cloud data of a target scene, and determining a target projection image corresponding to the target scene based on the point cloud data. Wherein, the target scene is also the traffic section of the road traffic sign to be extracted, and the content displayed by the target projection image at least includes the road traffic sign, which can be deployed on the road surface of the traffic section, and is a sign for managing traffic and ensuring safety, such as left turn sign, straight ahead sign, right turn sign, etc.

[0030] In an embodiment, the point cloud data of the target scene can be collected by a 3D camera, a laser radar and other devices with point cloud collection function in a downward-looking perspective. The point cloud data can be understood as a data set, and each point in the data set represents a set of position data and reflection intensity data. According to the coordinate data of each point in the point cloud data, the point is projected to a specified plane (such as XY plane), and the gray value of each pixel point in the specified plane is determined based on the reflection intensity data of each point, so as to obtain an initial projection image. The initial projection image is subjected to slice processing to obtain the target projection image.

[0031] Step S104, performing target detection processing on the target projection image by a pre-trained target detection model, to obtain sign attribute data corresponding to the road traffic sign. Wherein, the target detection model is obtained by distillation training of a lightweight detection model, the input of the target detection model is the target projection image, and the output is the sign attribute data corresponding to the road traffic sign, and the sign attribute data includes one or more of sign type, sign bounding box and sign angle.

[0032] In an implementation, the target projection image is input into the target detection model, and the target detection model outputs corresponding label attribute data.

[0033] In step S106, a target local image corresponding to the road traffic sign and a target template image are determined based on the label attribute data, and a label boundary corresponding to the road traffic sign is extracted from the target local image according to the target template image. The content displayed by the target local image includes the road traffic sign and the background, and the target template image is a template image of the road traffic sign. The template image is a single-channel image, the background pixel grayscale value is 0, the road traffic sign pixel grayscale value is 255, and the label boundary includes a plurality of boundary points.

[0034] In an implementation, a national standard size matching the size of the label bounding box can be determined, a target template image corresponding to the national standard size and the label type is found out, an initial local image is determined based on the size of the label bounding box, the initial local image is binarized in combination with the target template image to determine a target local image, and finally the label position and the label angle of the road traffic sign in the target local image are optimized according to the target template image to determine the boundary points of the road traffic sign, and then the label boundary corresponding to the road traffic sign is extracted from the target local image.

[0035] The extraction method of the road traffic sign provided by the embodiment of the present application first determines the target projection image corresponding to the target scene by using the point cloud data, then adopts a lightweight and simplified target detection model to perform target detection processing on the target projection image to obtain corresponding label attribute data by combining the advantages of the deep learning algorithm and the image post-processing algorithm, which not only reduces the requirement for the hardware device, but also effectively improves the machine recognition efficiency, and at the same time, the image post-processing algorithm is used to further optimize the machine recognition effect, the corresponding target local image and target template image are determined based on the label attribute data, and then the label boundary corresponding to the road traffic sign is extracted from the target local image by using the target template image, which not only guarantees the reliability of the recognition effect, but also meets the real-time requirement, and in addition, the embodiment of the present application also has a good recognition effect on the worn and blocked road traffic sign, and reduces the strong dependence of the traditional structure matching algorithm on the label features.

[0036] In order to facilitate the understanding of the foregoing embodiments, the foregoing step S102 is explained and described by the embodiment of the present application. After the point cloud data of the target scene is acquired, the step of determining the target projection image corresponding to the target scene based on the point cloud data can be performed according to the following (1) to (2):

[0037] (1) Based on the coordinate data of each point in the point cloud data, each point is projected onto a specified plane, and the gray value of each pixel point in the specified plane is determined based on the reflection intensity data of each point to obtain an initial projection image. In an embodiment, the point cloud data can be projected onto the specified plane according to the reflection intensity data, and the projection resolution is 0.25m*0.25m. Assuming that the specified plane is the XY plane, in actual application, there are multiple points in the point cloud data, and there are points in the point cloud data whose (X, Y) coordinates are the same and whose Z coordinates are different. Therefore, when the points are projected onto the XY plane, there can be a case that multiple points in the point cloud data are projected onto the same pixel point in the XY plane. Therefore, for each pixel point, if multiple points are projected onto the pixel point, the average value of the reflection intensity data of each point projected in the pixel point is taken as the gray value of the pixel point, and the initial projection image is obtained.

[0038] (2) The initial projection image is sliced to obtain a target projection image corresponding to the target scene. In an embodiment, the initial projection image is sliced according to a preset slicing step, for example, the slicing step is 300, and the size of the target projection image obtained by slicing is 512*512.

[0039] After obtaining the target projection image, it is necessary to use a lightweight model to identify the attribute data such as the sign type, the sign bounding box and the sign angle of the road traffic sign. In order to reduce the requirements for hardware devices and to improve the identification efficiency and effect, an embodiment of the present application provides a target detection model, and the construction process of the target detection model is shown in the following steps a to c:

[0040] Step a, obtaining a training data set. The training data set can include multiple traffic section images and labels of each traffic section image, and the labels include attribute data such as the sign type, the sign bounding box and the sign angle of each road traffic sign in the traffic section image.

[0041] Step b, training the pre-constructed initial detection model using the training data set, and distilling training the pre-constructed lightweight detection model using the trained initial detection model. The initial detection model can use an Oriented RCNN rotating detection network. In an embodiment, the training and identification of a complex model are first performed, specifically, the Oriented RCNN rotating detection network can be selected according to the task requirements to perform normal training, so as to obtain a high-precision initial detection model, and return the attribute data such as the sign type, the sign bounding box and the sign angle of the road sign; then the model structure is compressed, specifically, the road traffic sign data can be analyzed, a lightweight detection model is built, and the lightweight detection model is distilled and trained using the trained initial detection model, so as to obtain a model with similar precision.

[0042] Step c, the distilled lightweight detection model is compressed, and the format of the compressed lightweight detection model is converted to a specified format to obtain a target detection model. The specified format includes the ONNX format. In actual application, to further improve the running efficiency of the model, the model structure needs to be quantized to compress the size of the lightweight detection model. The precision of the compressed lightweight detection model remains at a comparable level. In addition, to be able to be applied, the model structure needs to be deployed. The compressed lightweight detection model can be converted into an ONNX structure by using the ONNX engine to obtain the target detection model. The target detection model can be applied to various hardware environments, thereby significantly improving the adaptability of the ONNX in different hardware environments.

[0043] To facilitate the understanding of the foregoing embodiments, the foregoing step S106 is explained and described in the embodiments of the present application. When the step of determining the target local image and the target template image corresponding to the road traffic sign based on the sign attribute data and extracting the sign boundary corresponding to the road traffic sign from the target local image according to the target template image is performed, the following steps 1 to 7 can be referred to:

[0044] Step 1, determining the national standard size corresponding to the road traffic sign according to the bounding box size of the sign bounding box, and determining the target template image corresponding to the road traffic sign based on the national standard size corresponding to the road traffic sign and the sign type. In actual application, a database for storing template images can be pre-configured. Since there may be certain errors in the calculation of the bounding box size of the sign bounding box, there is no template image consistent with the bounding box size in the database, so it is necessary to determine the national standard size corresponding to the road traffic sign according to the bounding box size of the sign bounding box, such as calculating the difference between the national standard size and the bounding box size of each template image in the database, and determining the national standard size with the smallest difference as the national standard size matched with the bounding box size, and then taking the national standard size matched with the bounding box size and the sign type as the retrieval condition to find the corresponding target template image in the database.

[0045] Step 2, determining the target local image corresponding to the road traffic sign based on the preset number of outwardly expanded pixels, the sign bounding box, the target projection image and the target template image. In specific implementation, the following steps 2.1 to 2.6 can be referred to:

[0046] Step 2.1, cropping the target projection image based on the preset number of outwardly expanded pixels and the sign bounding box to obtain an initial local image corresponding to the road traffic sign. For example, the preset number of outwardly expanded pixels can be set to 5 pixels, that is, the sign bounding box is expanded outwardly by 5 pixels to obtain a new sign bounding box, and the target projection image is cropped with the new sign bounding box as the cropping line to obtain the initial local image.

[0047] Step 2.2, determine the gray scale histogram of the initial local image. Wherein, the gray scale histogram is used to represent the frequency of occurrence of the gray scale value of each pixel point in the initial local image. In an embodiment, all pixel points in the initial local image can be counted according to the size of the gray scale value, and the frequency of occurrence is counted. The gray scale histogram is a function of the gray scale value, indicating the number of pixel points with a certain gray scale value in the initial local image, reflecting the frequency of occurrence of a certain gray scale value in the initial local image.

[0048] Step 2.3, according to the order from high to low of the gray scale value, the pixel number corresponding to the gray scale value in the gray scale histogram is added one by one until the pixel ratio obtained by adding is greater than the non-zero pixel ratio of the target template image, and the target gray scale value is determined. In an embodiment, the non-zero pixel ratio r of the target template image is first determined. Since the background pixel gray scale value in the template image is 0 and the traffic sign pixel gray scale value is 255, the non-zero pixel ratio r is also the ratio of the number of traffic sign pixels to the number of all pixels in the template image. Then the pixel number is added from high gray scale value to low gray scale value and the accumulated pixel ratio is calculated. When the accumulated pixel ratio is greater than the non-zero pixel ratio r, the addition is stopped, and the gray scale value at this time is determined as the target gray scale value K1. For example, assuming that the gray scale values in the gray scale histogram are in the order from high to low: x1, x2, x3, x4, if the sum of the pixel number corresponding to the gray scale value x1 and the pixel number corresponding to the gray scale value x2 is less than the non-zero pixel ratio r, the sum of the pixel number corresponding to the gray scale value x1, the pixel number corresponding to the gray scale value x2 and the pixel number corresponding to the gray scale value x3 is calculated, and if the sum is greater than the non-zero pixel ratio r at this time, the gray scale value x3 is determined as the target gray scale value K1.

[0049] Step 2.4, determine the segmentation threshold value corresponding to the initial local image by using the Otsu algorithm. Wherein, the Otsu algorithm is used to degrade the initial local image from a gray scale image to a binary image. The Otsu algorithm assumes that the initial local image contains two types of pixels according to the bimodal histogram (traffic sign pixels and background pixels), and calculates the best threshold value that can separate the two types of pixels, so that the intra-class variance between the two types of pixels is minimized. The best threshold value is denoted as the segmentation threshold value K2.

[0050] Step 2.5, if the target gray value is greater than the segmentation threshold value, the target gray value is determined as the binarization threshold value; or if the segmentation threshold value is greater than the target gray value, the segmentation threshold value is determined as the binarization threshold value. In an embodiment, the larger value between the target gray value K1 and the segmentation threshold value K2 can be taken as the final binarization threshold value K. For example, if the target gray value K1 is greater than the segmentation threshold value K2, the target gray value K1 is the binarization threshold value K; otherwise, if the target gray value K1 is less than the segmentation threshold value K2, the segmentation threshold value K2 is the binarization threshold value K.

[0051] Step 2.6, binarizing the initial local image using the binarization threshold value to obtain a target local image corresponding to the road traffic sign.

[0052] Step 3, cutting the preset angle threshold range using a preset step size to obtain a plurality of angle threshold values. For example, assuming that the preset step size is 0.1 degree and the preset angle threshold range is -10 degrees to +10 degrees, the preset angle threshold range can be cut into 201 angle threshold values.

[0053] Step 4, superimposing the sign angle corresponding to the road traffic sign with each angle threshold value respectively to obtain a plurality of candidate rotation angles. For example, calculating the sum value between the sign angle and the angle threshold value 0.1 degree to obtain a candidate rotation angle, calculating the sum value between the sign angle and the angle threshold value -0.1 degree to obtain another candidate rotation angle, and a total of 201 candidate rotation angles can be calculated.

[0054] Step 5, for each candidate rotation angle, rotating the target local image according to the candidate rotation angle, determining the difference degree between the rotated target local image and the target template image, and determining the sign position under the candidate rotation angle based on the difference degree. In specific implementation, reference can be made to the following steps 5.1 to 5.3:

[0055] Step 5.1, adjusting the size of the target template image according to the size of the outer bounding box of the sign to obtain an adjusted template image. In an embodiment, the size of the target template image can be adjusted to the size of the outer bounding box, so that the size of the adjusted template image is consistent with that of the target local image, and thus the difference degree between the adjusted template image and the target local image can be better determined.

[0056] Step 5.2, determining the first difference degree between the rotated target local image and the target template image and the first sign position of the road traffic sign in the target local image through pixel-by-pixel translation comparison; and determining the second difference degree between the rotated target local image and the adjusted template image and the second sign position of the road traffic sign in the target local image through pixel-by-pixel translation comparison. In an embodiment, the difference degree can be determined according to the following formula:

[0057]

[0058] wherein, R is the difference degree, T is the rotated template image, I is the target local image, m and n are the row number and column number of the target template image respectively.

[0059] Step 5.3, if the first difference degree is less than the second difference degree, the first mark position is determined as the mark position under the candidate rotation angle; or if the first difference degree is greater than the second difference degree, the second mark position is determined as the mark position under the candidate rotation angle. In an embodiment, the mark position corresponding to the minimum difference degree can be selected as the mark position under the candidate rotation angle.

[0060] In another embodiment, the size of the rotated target local image can be adjusted according to the national standard size to obtain an adjusted local image; a third difference degree between the adjusted local image and the target template image and a third mark position of the road traffic sign in the target local image are determined through pixel-by-pixel translation comparison; and a fourth difference degree between the adjusted local image and the target template image and a fourth mark position of the road traffic sign in the target local image are determined through pixel-by-pixel translation comparison; if the third difference degree is less than the fourth difference degree, the third mark position is determined as the mark position under the candidate rotation angle; or if the third difference degree is greater than the fourth difference degree, the fourth mark position is determined as the mark position under the candidate rotation angle.

[0061] Step 6, the target rotation angle is determined from each candidate rotation angle according to the difference degree corresponding to each candidate rotation angle, and the mark position under the target rotation angle is determined as the target mark position. In an embodiment, the candidate rotation angle corresponding to the minimum difference degree can be determined as the target rotation angle, and the mark position under the target rotation angle is the target mark position.

[0062] Step 7, the mark boundary corresponding to the road traffic sign is extracted from the target local image based on the target rotation angle and the target mark position. The mark boundary includes a plurality of boundary points. In an embodiment, the boundary points of the road traffic sign in the target local image are extracted according to the obtained target rotation angle and target mark position, and the boundary points of the road traffic sign are saved.

[0063] In order to facilitate the understanding of the foregoing embodiments, the application provides an application example of the road traffic sign extraction method, which is described below with reference to Figure 2 Another flowchart of the road traffic sign extraction method is shown in FIG. 2, which mainly includes the following steps S202 to S214:

[0064] Step S202: Obtain point cloud data of the target scene.

[0065] Step S204: Determine the initial projected image based on the point cloud data.

[0066] Step S206: Slice the initial projection image to obtain the target projection image.

[0067] Step S208: Identify the sign attribute data of road traffic signs, such as sign type, sign outer frame, and sign angle, through a lightweight detection model.

[0068] Step S210: Determine the target local image based on the marker attribute data.

[0069] Step S212: Optimize the sign angle and sign position of the road traffic signs in the target local image.

[0070] Step S214: Determine the boundary of the road traffic sign based on the sign angle and sign position. For example, see... Figure 3 The diagram shown illustrates the extraction effect of road traffic signs in a large-scale scene. Figure 4 The image shown is a detailed representation of the extraction effect of a road traffic sign. Figure 5 This diagram illustrates the extraction effect of road traffic signs in a worn or missing scenario. (See above.) Figures 3 to 5 The method for extracting road traffic signs provided in this invention can automatically and quickly extract traffic signs in large scenes, with an extraction rate of up to 99%, which is 20% more efficient than traditional deep learning methods. It also improves extraction accuracy by 30% compared to traditional structure matching methods. Furthermore, it can achieve an 80% extraction rate for occluded or worn traffic signs.

[0071] In summary, the method for extracting road traffic signs provided by this invention combines deep learning recognition algorithms and image post-processing algorithms, solving the problems of low extraction efficiency of traditional deep learning algorithms and over-reliance on sign features in the recognition effect of traditional structure matching algorithms. This method requires no preprocessing of point cloud data and image data, exhibits good robustness, and can achieve fully automatic traffic sign extraction without manual intervention.

[0072] Regarding the method for extracting road traffic signs provided in the foregoing embodiments, this invention provides a device for extracting road traffic signs, see [link to relevant documentation]. Figure 6 The diagram shows a structural schematic of a road traffic sign extraction device, which mainly includes the following parts:

[0073] The projection image determination module 602 is configured to acquire point cloud data of a target scene, and determine a target projection image corresponding to the target scene based on the point cloud data, wherein the target projection image displays at least a road traffic sign;

[0074] The sign detection module 604 is configured to perform target detection processing on the target projection image by using a pre-trained target detection model to obtain sign attribute data corresponding to the road traffic sign, wherein the target detection model is obtained by distillation training on a lightweight detection model.

[0075] The sign extraction module 606 is configured to determine a target local image and a target template image corresponding to the road traffic sign based on the sign attribute data, and extract a sign boundary corresponding to the road traffic sign from the target local image according to the target template image.

[0076] The extraction device for the road traffic sign provided by the embodiment of the present application first determines a target projection image corresponding to a target scene by using point cloud data, and then combines the advantages of the deep learning algorithm and the image post-processing algorithm, adopts a lightweight and simplified target detection model to perform target detection processing on the target projection image to obtain corresponding sign attribute data, which can not only reduce the requirement for hardware devices, but also effectively improve the machine recognition efficiency, and at the same time, the image post-processing algorithm is used to further optimize the machine recognition effect, the corresponding target local image and target template image are determined based on the sign attribute data, and then the target template image is used to extract the sign boundary corresponding to the road traffic sign from the target local image, which not only ensures the reliability of the recognition effect, but also meets the real-time requirement, in addition, the embodiment of the present application also has a good recognition effect on the worn and blocked road traffic sign, and reduces the strong dependence of the traditional structure matching algorithm on the sign features.

[0077] In one embodiment, the above device further includes a training module configured to: acquire a training data set; train an initial detection model pre-constructed by using the training data set, and perform distillation training on a lightweight detection model pre-constructed by using the initial detection model after training; perform compression processing on the lightweight detection model after distillation training, and convert the format of the lightweight detection model after compression processing to a specified format to obtain the target detection model; wherein the specified format includes an ONNX format.

[0078] In one embodiment, the sign attribute data includes a sign type and a sign bounding box, and the sign extraction module 606 is further configured to: determine a national standard size corresponding to the road traffic sign according to the size of the sign bounding box, and determine a target template image corresponding to the road traffic sign based on the national standard size corresponding to the road traffic sign and the sign type; and determine a target local image corresponding to the road traffic sign based on a preset number of outward expansion pixels, the sign bounding box, the target projection image and the target template image.

[0079] In an embodiment, the sign extraction module 606 is further configured to: crop the target projection image based on the preset number of outer expanded pixels and the sign bounding box to obtain an initial local image corresponding to the road traffic sign; determine a gray histogram of the initial local image; in order from high to low, accumulate the number of pixels corresponding to each gray value in the gray histogram until the accumulated pixel ratio is greater than the non-zero pixel ratio of the target template image, and determine a target gray value; determine a segmentation threshold value corresponding to the initial local image using the Otsu algorithm; if the target gray value is greater than the segmentation threshold value, determine the target gray value as a binarization threshold value; or if the segmentation threshold value is greater than the target gray value, determine the segmentation threshold value as the binarization threshold value; and perform binarization processing on the initial local image using the binarization threshold value to obtain a target local image corresponding to the road traffic sign.

[0080] In an embodiment, the sign attribute data further includes a sign angle; and the sign extraction module 606 is further configured to: cut the preset angle threshold range using a preset step size to obtain a plurality of angle threshold values; superimpose the sign angle corresponding to the road traffic sign with each angle threshold value to obtain a plurality of candidate rotation angles; for each candidate rotation angle, rotate the target local image according to the candidate rotation angle, determine a difference degree between the rotated target local image and the target template image, and determine a sign position under the candidate rotation angle based on the difference degree; determine a target rotation angle from each candidate rotation angle according to the difference degree corresponding to each candidate rotation angle, and determine the sign position under the target rotation angle as a target sign position; and extract a sign boundary corresponding to the road traffic sign from the target local image based on the target rotation angle and the target sign position.

[0081] In an embodiment, the sign extraction module 606 is further configured to: adjust the size of the target template image to obtain an adjusted template image according to the size of the bounding box of the sign bounding box; determine a first difference degree between the rotated target local image and the target template image, and a first sign position of the road traffic sign in the target local image by pixel-by-pixel translation comparison; and determine a second difference degree between the rotated target local image and the adjusted template image, and a second sign position of the road traffic sign in the target local image by pixel-by-pixel translation comparison; if the first difference degree is less than the second difference degree, determine the first sign position as the sign position under the candidate rotation angle; or if the first difference degree is greater than the second difference degree, determine the second sign position as the sign position under the candidate rotation angle.

[0082] In an implementation, the projection image determination module 602 is further configured to: project each point in the point cloud data onto a specified plane based on coordinate data of each point, and determine a gray value of each pixel in the specified plane based on the reflection intensity data of each point to obtain an initial projection image; and perform slice processing on the initial projection image to obtain a target projection image corresponding to the target scene.

[0083] The device provided by the embodiments of the present application has the same implementation principle and technical effects as the foregoing method embodiments. For brevity, the part not mentioned in the device embodiment part can be referred to the corresponding content in the foregoing method embodiments.

[0084] The electronic device provided by the embodiments of the present application includes a processor and a storage device. The storage device stores a computer program. When the computer program is run by the processor, the method according to any one of the embodiments described above is executed.

[0085] Figure 7 The electronic device 100 includes a processor 70, a memory 71, a bus 72 and a communication interface 73. The processor 70, the communication interface 73 and the memory 71 are connected through the bus 72. The processor 70 is configured to execute executable modules stored in the memory 71, such as a computer program.

[0086] The memory 71 can include a high-speed random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. The communication between the system network element and at least one other network element is realized through at least one communication interface 73 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0087] The bus 72 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of brevity, Figure 7 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0088] The memory 71 is configured to store a program. After receiving an execution instruction, the processor 70 executes the program. The method executed by the device defined by the flow process disclosed in any one of the foregoing embodiments of the present application can be applied to the processor 70 or realized by the processor 70.

[0089] The processor 70 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuit of hardware in the processor 70 or by instructions in the form of software. The processor 70 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 71, and the processor 70 reads the information in the memory 71, and combines the hardware to complete the steps of the above method.

[0090] The computer program product of the readable storage medium provided by the embodiments of the present application comprises a computer readable storage medium storing program codes, and the program codes comprise instructions for executing the method described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be described here.

[0091] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0092] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of extracting a road traffic sign, characterized by, The method comprises: acquiring point cloud data of a target scene, and determining a target projection image corresponding to the target scene based on the point cloud data; wherein the target projection image displays at least a road traffic sign; performing target detection processing on the target projection image by using a pre-trained target detection model to obtain sign attribute data corresponding to the road traffic sign; wherein the target detection model is obtained by distilling a lightweight detection model, and the sign attribute data comprises a sign angle; based on the sign attribute data, determining a target local image and a target template image corresponding to the road traffic sign, and extracting a sign boundary corresponding to the road traffic sign from the target local image according to the target template image; extracting the sign boundary corresponding to the road traffic sign from the target local image according to the target template image comprises: cutting a preset angle threshold range by using a preset step to obtain a plurality of angle thresholds; superimposing the sign angle corresponding to the road traffic sign on each of the angle thresholds to obtain a plurality of candidate rotation angles; for each candidate rotation angle, rotating the target local image according to the candidate rotation angle, determining a difference degree between the rotated target local image and the target template image, and determining a sign position under the candidate rotation angle based on the difference degree; determining a target rotation angle from each candidate rotation angle according to the difference degree corresponding to each candidate rotation angle, and determining a target sign position under the target rotation angle; extracting the sign boundary corresponding to the road traffic sign from the target local image based on the target rotation angle and the target sign position.

2. The method of claim 1, wherein The method further comprises: acquiring a training data set; training a pre-constructed initial detection model by using the training data set, and distilling a pre-constructed lightweight detection model by using the trained initial detection model; performing compression processing on the distilled lightweight detection model, and converting the format of the compressed lightweight detection model to a specified format to obtain a target detection model; wherein the specified format comprises an ONNX format.

3. The method of claim 1, wherein The sign attribute data comprises a sign type and a sign bounding box; based on the sign attribute data, determining a target local image and a target template image corresponding to the road traffic sign comprises: determining a national standard size corresponding to the road traffic sign according to the size of the bounding box of the sign bounding box, and determining a target template image corresponding to the road traffic sign based on the national standard size corresponding to the road traffic sign and the sign type; determining a target local image corresponding to the road traffic sign based on a preset number of outwardly expanded pixels, the sign bounding box, the target projection image and the target template image.

4. The method of claim 3, wherein determining a target local image corresponding to the road traffic sign based on a preset number of outwardly expanded pixels, the sign bounding box, the target projection image and the target template image comprises: carving the target projection image based on a preset number of outwardly expanded pixels and the sign bounding box to obtain an initial local image corresponding to the road traffic sign; determining a gray histogram of the initial local image; in order of descending gray value, adding the number of pixels corresponding to each gray value in the gray histogram one by one until the pixel ratio obtained by adding is greater than the non-zero pixel ratio of the target template image, and determining a target gray value; determining a segmentation threshold value corresponding to the initial local image using the Otsu algorithm; if the target gray value is greater than the segmentation threshold value, determining the target gray value as a binarization threshold value; or if the segmentation threshold value is greater than the target gray value, determining the segmentation threshold value as the binarization threshold value; performing binarization processing on the initial local image using the binarization threshold value to obtain a target local image corresponding to the road traffic sign.

5. The method of claim 1, wherein determining a difference degree between the target local image after rotation and the target template image, and determining a sign position under the candidate rotation angle based on the difference degree, including: adjusting the size of the target template image to obtain an adjusted template image according to the size of the bounding box of the sign bounding box; determining a first difference degree between the target local image after rotation and the target template image and a first sign position of the road traffic sign in the target local image through pixel-by-pixel translation comparison; and determining a second difference degree between the target local image after rotation and the adjusted template image and a second sign position of the road traffic sign in the target local image through pixel-by-pixel translation comparison; if the first difference degree is less than the second difference degree, determining the first sign position as the sign position under the candidate rotation angle; or if the first difference degree is greater than the second difference degree, determining the second sign position as the sign position under the candidate rotation angle.

6. The method of claim 1, wherein determining a target projection image corresponding to the target scene based on the point cloud data, including: projecting each point in the point cloud data onto a specified plane based on the coordinate data of each point, and determining the gray value of each pixel point in the specified plane based on the reflection intensity data of each point to obtain an initial projection image; performing slice processing on the initial projection image to obtain a target projection image corresponding to the target scene.

7. An apparatus for extracting a road traffic sign, characterized by comprising: including: a projection image determination module configured to obtain point cloud data of a target scene and determine a target projection image corresponding to the target scene based on the point cloud data; wherein the target projection image displays at least a road traffic sign; a sign detection module configured to perform target detection processing on the target projection image by using a pre-trained target detection model to obtain sign attribute data corresponding to the road traffic sign; wherein the target detection model is obtained by distilling training a lightweight detection model, and the sign attribute data includes a sign angle; The sign extraction module is configured to determine a target local image and a target template image corresponding to the road traffic sign based on the sign attribute data, and extract a sign boundary corresponding to the road traffic sign from the target local image according to the target template image. The sign extraction module is specifically configured to: cut a preset angle threshold range by using a preset step length to obtain a plurality of angle thresholds; superimpose the sign angle corresponding to the road traffic sign with each of the angle thresholds to obtain a plurality of candidate rotation angles; for each of the candidate rotation angles, rotate the target local image according to the candidate rotation angle, determine a difference degree between the rotated target local image and the target template image, and determine a sign position under the candidate rotation angle based on the difference degree; determine a target rotation angle from each of the candidate rotation angles according to the difference degree corresponding to each of the candidate rotation angles, and determine the sign position under the target rotation angle as a target sign position; extract a sign boundary corresponding to the road traffic sign from the target local image based on the target rotation angle and the target sign position.

8. An electronic device, comprising: A processor and a memory are included, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to implement the method in any one of claims 1 to 6.

Images