Traffic sign board positioning method based on color features and depth image features

Abstract

The invention discloses a traffic sign positioning method based on color features and depth image features, and relates to the technical field of road asset intelligent inventory image processing, and the method comprises the steps: constructing HSV color space data containing a hue component, a saturation component and a brightness component according to a panoramic image, carrying out the color segmentation processing of each component, and obtaining a target image; obtaining a traffic sign board region of interest; performing coordinate transformation by using the initial transformation parameter, generating a corresponding panoramic depth image by combining exterior orientation elements of each panoramic image, extracting a target point cloud set in a region of interest of the traffic sign board, and determining a coordinate position of the traffic sign board by using a quantitative feature evaluation rule; and extracting a fusion feature vector through a local binary pattern algorithm and a histogram of oriented gradient algorithm, constructing a traffic sign classification model, inputting the coordinate position of the traffic sign board, and outputting the category information of the traffic sign board. According to the invention, high-precision identification of traffic signs in a complex traffic scene is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing for intelligent inventory of highway assets, and in particular to a method for locating traffic signs based on color features and depth image features. Background Art

[0002] Signboards are the basic infrastructure on highways to ensure driving safety and guide traffic flow. With the continuous increase in the scale of road networks and the rapid development of intelligent transportation systems, the number of signboards has increased dramatically. As a highway asset management and operation unit, we are faced with a huge number of signs. Our management level not only affects road safety and traffic efficiency, but also affects our operation and maintenance costs.

[0003] Traffic sign automatic recognition technology plays an important role in road safety management, autonomous driving and low-altitude economic construction. Under the concept of full life cycle management of highway assets, the inspection, maintenance and updating of signs are becoming more frequent. In the existing highway asset management system, the maintenance of signs mainly relies on manual periodic inspection and recording. This traditional method has defects such as low efficiency and inability to respond in real time, and it is difficult to meet the growing needs of traffic safety management; therefore, the use of intelligent technology to identify and locate signs can greatly improve management efficiency and operation and maintenance level.

[0004] In recent years, with the development of computer vision technology and deep learning algorithms, traffic sign recognition methods based on image processing have made significant progress and can intelligently provide the location and status information of signboards. However, the recognition accuracy and positioning accuracy in complex environments still face many challenges.

[0005] At present, traffic sign recognition mainly adopts a single feature extraction method based on image processing. Among them, the color feature-based method extracts the sign area through RGB color space analysis, but it is easily affected by lighting changes and weather conditions; the edge feature-based method recognizes standard shapes by detecting geometric edges in the image, but it is easily disturbed in complex backgrounds; the texture feature-based method recognizes signs by analyzing the local texture pattern of the image, but the recognition effect is poor for blurred or partially occluded signs; in addition, some studies have begun to try to use deep learning methods to directly extract features from images, but such methods often require a large amount of labeled data for training, and the computational complexity is high, making it difficult to meet the needs of real-time processing.

[0006] However, the existing traffic sign recognition methods still have some obvious shortcomings. First, most methods rely only on single feature information for recognition, and fail to effectively integrate the advantages of multiple features, resulting in unstable recognition performance in complex environments; second, existing methods generally lack the use of three-dimensional spatial information, and only rely on two-dimensional image features for recognition, and cannot accurately obtain the actual spatial position and geometric features of the sign; in addition, in panoramic road images, due to the changing shooting angles and high scene complexity, the existing traffic sign recognition algorithms based on machine learning often need to extract features and classify a large number of candidate areas, which is not only computationally inefficient, but also easily affected by background interference, leading to misrecognition, and it is difficult to meet the requirements of recognition accuracy and processing efficiency in practical applications.

[0007] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0008] In response to the problems in the related technology, the present invention proposes a traffic sign positioning method based on color features and depth image features. It has the advantage of utilizing the high-precision three-dimensional spatial information characteristics of point cloud data to perform feature matching and spatial positioning analysis on traffic signs in panoramic images, thereby providing complete data support for the precise extraction of corresponding target positions in the image, thereby solving the problem in the prior art that the extraction of traffic signs in road images based on machine learning algorithms is not efficient and accurate enough.

[0009] To this end, the specific technical solution adopted by the present invention is as follows:

[0010] A traffic sign positioning method based on color features and depth image features, the traffic sign positioning method based on color features and depth image features comprises the following steps:

[0011] S1. Based on the RGB color space data of the panoramic image, HSV color space data including hue component, saturation component and brightness component is constructed, and the threshold range of each component is determined. By performing color segmentation processing on each component of the HSV color space data, the region of interest of the traffic sign is obtained;

[0012] S2. Based on the point cloud data in the laser radar coordinate system, the initial transformation parameters are used to perform coordinate transformation, and the corresponding panoramic depth image is generated by combining the exterior orientation elements of each panoramic image. The target point cloud collection in the area of ​​interest of the traffic sign is extracted, and the coordinate position of the traffic sign in the panoramic image is determined by using quantitative feature evaluation rules;

[0013] S3. Based on the collected panoramic image data, a traffic sign sample library is established, and the fused feature vector is extracted through the local binary pattern algorithm and the directional gradient histogram algorithm to construct a traffic sign classification model; the traffic sign classification model is trained using the traffic sign sample library, the coordinate position of the traffic sign in the panoramic image is input, and the category information of the traffic sign is output.

[0014] Furthermore, based on the RGB color space data of the panoramic image, HSV color space data including hue component, saturation component and brightness component is constructed, and the threshold range of each component is determined. By performing color segmentation processing on each component of the HSV color space data, the region of interest of the traffic sign is obtained, which includes the following steps:

[0015] S11, calculating a color space difference based on the RGB color space data of the panoramic image, and generating HSV color space data according to the color space difference, wherein the HSV color space data includes a hue component, a saturation component, and a brightness component;

[0016] S12, extracting the color pixels of the traffic sign in the panoramic image according to the hue component, the saturation component and the brightness component, and determining the threshold range of each component by using the threshold adjustment function, performing color segmentation processing on each component of the HSV color space data by AND operation, and obtaining a color segmentation image;

[0017] S13. Use median filtering technology to remove noise in the color segmentation image and obtain the region of interest of the traffic sign.

[0018] Furthermore, based on the RGB color space data of the panoramic image, calculating the color space difference, and generating the HSV color space data according to the color space difference includes the following steps:

[0019] S111, standardizing the RGB color space data of the panoramic image to obtain standard color components;

[0020] S112, calculating the maximum and minimum values ​​of the standard color components, and solving the color space difference;

[0021] S113. Generate a hue component, a saturation component, and a brightness component based on the color space difference to obtain HSV color space data.

[0022] Furthermore, the expression of standard color components is:

[0023]

[0024] Where R, G, and B are the red, green, and blue color components of the panoramic image, respectively; R′, G′, and B′ are the red, green, and blue color components of the standard color components, respectively;

[0025] The expression of color space difference is:

[0026]

[0027]

[0028] In the formula, I max is the maximum value among the standard color components, I min is the minimum value among the standard color components, and δ is the color space difference;

[0029] The expression of HSV color space data is:

[0030]

[0031]

[0032]

[0033] Wherein, H is the hue component of the HSV color space data, S is the saturation component of the HSV color space data, and V is the brightness component of the HSV color space data.

[0034] Furthermore, based on the point cloud data in the laser radar coordinate system, the coordinate transformation is performed using the initial transformation parameters, and the corresponding panoramic depth image is generated by combining the exterior orientation elements of each panoramic image, and the target point cloud collection in the area of ​​interest of the traffic sign is extracted. The coordinate position of the traffic sign in the panoramic image is determined using the quantitative feature evaluation rule, including the following steps:

[0035] S21, performing coordinate transformation using initial transformation parameters according to the point cloud data in the laser radar coordinate system to obtain the point cloud data in the panoramic camera coordinate system, and solving the first characteristic angle and the second characteristic angle of the line connecting the spatial point and the camera center, combining the exterior orientation elements of each panoramic image, and generating a corresponding panoramic depth image;

[0036] S22, based on the panoramic depth image corresponding to each panoramic image, extracting the point cloud data falling into the area of ​​interest of the traffic sign, and establishing a target point cloud collection;

[0037] S23. Use quantitative feature evaluation rules to evaluate the matching degree of data points in the target point cloud collection, and determine the coordinate position of the traffic sign in the panoramic image based on the matching degree evaluation results of the data points, wherein the quantitative feature evaluation rules include height feature evaluation, shape feature evaluation and continuity feature evaluation.

[0038] Further, according to the point cloud data in the laser radar coordinate system, coordinate transformation is performed using the initial transformation parameters to obtain the point cloud data in the panoramic camera coordinate system, and the first characteristic angle and the second characteristic angle of the line connecting the space point and the camera center are solved, and the exterior orientation elements of each panoramic image are combined to generate the corresponding panoramic depth image, including the following steps:

[0039] S211, solving initial conversion parameters according to the point cloud data in the laser radar coordinate system, and using the initial conversion parameters to perform coordinate conversion to obtain point cloud data in the panoramic camera coordinate system, wherein the initial conversion parameters include a rotation matrix and a translation vector;

[0040] S212, based on the point cloud data in the panoramic camera coordinate system, obtaining the point cloud data in the spherical coordinate system by calculating the first characteristic angle and the second characteristic angle of the line connecting the spatial point and the camera center;

[0041] S213, determining the point cloud data in the pixel coordinate system based on the point cloud data in the spherical coordinate system and combining the exterior orientation elements of each panoramic image, and generating a corresponding panoramic depth image, wherein the exterior orientation elements of each panoramic image include the width and height of each panoramic image.

[0042] Furthermore, the expression of the point cloud data in the panoramic camera coordinate system is:

[0043]

[0044] In the formula, X c , Y c , Z c They are the point cloud data of the X-axis, Y-axis, and Z-axis in the panoramic camera coordinate system, R is the rotation matrix between the laser radar coordinate system and the panoramic camera coordinate system, X w , Y w , Z w are the point cloud data of the X-axis, Y-axis, and Z-axis in the laser radar coordinate system, and T is the translation vector between the laser radar coordinate system and the panoramic camera coordinate system;

[0045] The point cloud data in the spherical coordinate system includes the first characteristic angle and the second characteristic angle. The expressions of the first characteristic angle and the second characteristic angle are:

[0046]

[0047] In the formula, is the angle between the line connecting the space point and the camera center and the Z axis, and θ is the angle between the projection of the line connecting the space point and the camera center on the XY plane and the X axis;

[0048] The expression of point cloud data in pixel coordinate system is:

[0049]

[0050] Where u and v are the horizontal pixel coordinates and vertical pixel coordinates in the panoramic depth image, respectively, and H and W are the height and width of the panoramic image, respectively.

[0051] Furthermore, the height feature evaluation includes calculating the height of the target area based on the panoramic image, and excluding the traffic sign interest area whose maximum value of the Y axis is greater than the height of the target area;

[0052] Shape feature assessment includes calculating the center point coordinates of all points in the traffic sign's area of ​​interest, the length of the line between the corner points, and the distance from the center point to any boundary point based on the target point cloud collection, and assessing whether the target area meets the shape features of circular, rectangular, and equilateral triangle signs;

[0053] The continuity feature evaluation includes plane fitting based on the target point cloud collection through the RANSAC algorithm, calculating the distance from the point to the plane and the plane equation coefficient, and evaluating the plane continuity and mutation of the target area.

[0054] Furthermore, the expression of the target area height is:

[0055]

[0056] Where X and Y are the X-axis coordinate and Y-axis coordinate of the upper left corner of the target area, respectively; w and h are the width and height of the target area, respectively; w in With h in are the width and height of the panoramic image respectively;

[0057] The expression of the center point coordinates of all points in the traffic sign area of ​​interest is:

[0058]

[0059] In the formula, x c ,y c 、z c are the X-axis coordinate, Y-axis coordinate and Z-axis coordinate of the center point respectively, n is the number of all points in the area of ​​interest of the traffic sign, x i ,y i 、z i are the X-axis coordinate, Y-axis coordinate, and Z-axis coordinate of the i-th point in the target point cloud collection.

[0060] Furthermore, a traffic sign sample library is established based on the collected panoramic image data, and a fusion feature vector is extracted through a local binary pattern algorithm and a directional gradient histogram algorithm to construct a traffic sign classification model; the traffic sign classification model is trained using the traffic sign sample library, the coordinate position of the traffic sign in the panoramic image is input, and the category information of the traffic sign is output, including the following steps:

[0061] S31. Based on the collected panoramic image data, a sign area is selected as a positive sample, and a non-sign area is selected as a negative sample to obtain a traffic sign sample library;

[0062] S32. Extract texture feature vectors using a local binary pattern algorithm, extract shape feature vectors using a directional gradient histogram algorithm, and use a principal component analysis method to reduce the dimension of the shape feature vectors to obtain a fused feature vector and establish a feature extraction network; based on the feature extraction network and combined with a one-to-many classification strategy, construct a traffic sign classification model;

[0063] S33. Train a traffic sign classification model using a traffic sign sample library, and based on the trained traffic sign classification model, input the coordinate position of the traffic sign in the panoramic image, and output the category information of the traffic sign.

[0064] The beneficial effects of the present invention are:

[0065] (1) The present invention decomposes positioning and category judgment into two independent steps. It uses color features and depth information features to accurately locate traffic signs. On the basis of accurate positioning, it uses a classification model to distinguish the category of the sign in the positioning area, thereby achieving high-precision recognition of traffic signs in complex traffic scenes.

[0066] (2) The present invention utilizes the distinct color characteristics of traffic signs in panoramic images and adopts HSV color segmentation to achieve preliminary positioning of the sign area. The method is simple in principle, requires little calculation, and is easy to implement.

[0067] (3) The present invention proposes a feature extraction method that integrates texture information and shape information. The HOG feature is difficult to present the overall texture of traffic signs and is prone to errors when there is slight occlusion, while the LBP feature is good at describing texture and is not afraid of slight deformation and occlusion. The present invention combines the two to complement each other's advantages, significantly improves the recognition accuracy, and enhances the performance and stability of the system in complex traffic scenes.

[0068] (4) The present invention addresses the problem that the extraction of traffic signs from road images based on machine learning algorithms is not efficient and accurate enough. By utilizing the high-precision three-dimensional spatial information characteristics of point cloud data, the present invention performs feature matching and spatial positioning analysis on traffic signs in panoramic images, thereby providing complete data support for the accurate extraction of corresponding target positions in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 It is a flow chart of a method for locating a traffic sign based on color features and depth image features according to an embodiment of the present invention;

[0071] Figure 2 An HSV color space segmentation image obtained by a traffic sign positioning method based on color features and depth image features according to an embodiment of the present invention;

[0072] Figure 3 A panoramic depth image corresponding to the panoramic image obtained by a traffic sign positioning method based on color features and depth image features according to an embodiment of the present invention;

[0073] Figure 4 It is a schematic diagram of a feature fusion process in a traffic sign positioning method based on color features and depth image features according to an embodiment of the present invention;

[0074] Figure 5 It is a schematic diagram of the accuracy and running time corresponding to different feature fusion strategies in a traffic sign positioning method based on color features and depth image features according to an embodiment of the present invention;

[0075] Figure 6 It is a specific implementation diagram of traffic sign positioning based on color features and depth image features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0076] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0077] According to an embodiment of the present invention, a traffic sign positioning method based on color features and depth image features is provided.

[0078] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 and Figure 6As shown, according to an embodiment of the present invention, a method for locating a traffic sign based on color features and depth image features is provided. The method for locating a traffic sign based on color features and depth image features comprises the following steps:

[0079] S1. Based on the RGB color space data of the panoramic image, HSV color space data including hue component, saturation component and brightness component is constructed, and the threshold range of each component is determined. By performing color segmentation processing on each component of the HSV color space data, the region of interest of the traffic sign is obtained;

[0080] S2. Based on the point cloud data in the laser radar coordinate system, the initial transformation parameters are used to perform coordinate transformation, and the corresponding panoramic depth image is generated by combining the exterior orientation elements of each panoramic image. The target point cloud collection in the area of ​​interest of the traffic sign is extracted, and the coordinate position of the traffic sign in the panoramic image is determined by using quantitative feature evaluation rules;

[0081] S3. Based on the collected panoramic image data, a traffic sign sample library is established, and the fused feature vector is extracted through the local binary pattern algorithm and the directional gradient histogram algorithm to construct a traffic sign classification model; the traffic sign classification model is trained using the traffic sign sample library, the coordinate position of the traffic sign in the panoramic image is input, and the category information of the traffic sign is output.

[0082] Specifically, the present invention proposes a method for automatically identifying traffic signs that combines color features and depth features, decomposing positioning and category judgment into two independent steps, making full use of the spatial geometric information in the point cloud data and the semantic features in the panoramic image to improve the recognition accuracy of the sign. First, the region of interest of the traffic sign is extracted from the panoramic image using HSV color segmentation; then, the laser point cloud is projected according to the external orientation parameters of the camera to generate a corresponding depth image for each panoramic image; using the panoramic depth image as a medium, the point cloud data falling into the region of interest is extracted respectively, and the key features such as height, shape, continuity and mutation provided by these point cloud data are used to evaluate whether the region actually contains the target traffic sign, and the wrong region of interest is filtered out, so as to accurately locate the actual position of the sign; for the sign area that has been accurately located, a deep learning model that combines texture features and shape features is used to perform category judgment; finally, the category and location information of the traffic sign are combined to realize the automatic recognition of the traffic sign in the panoramic image.

[0083] In one embodiment, based on the RGB color space data of the panoramic image, HSV color space data including hue component, saturation component and brightness component is constructed, and the threshold range of each component is determined. By performing color segmentation processing on each component of the HSV color space data, the region of interest of the traffic sign is obtained, which includes the following steps:

[0084] S11, calculating a color space difference based on the RGB color space data of the panoramic image, and generating HSV color space data according to the color space difference, wherein the HSV color space data includes a hue component, a saturation component, and a brightness component;

[0085] S12, extracting the color pixels of the traffic sign in the panoramic image according to the hue component, the saturation component and the brightness component, and determining the threshold range of each component by using the threshold adjustment function, performing color segmentation processing on each component of the HSV color space data by AND operation, and obtaining a color segmentation image;

[0086] S13. Use median filtering technology to remove noise in the color segmentation image and obtain the region of interest of the traffic sign.

[0087] Specifically, the panoramic camera and lidar carried by the vehicle-mounted mobile system are used to collect panoramic image data and point cloud data respectively.

[0088] Specifically, in step S1, based on the collected panoramic image data, HSV color threshold segmentation is used to extract the region of interest (ROI) containing the traffic sign. The principle of this method is to filter out pixels that are different from the background color of the sign from the original image, and retain only pixels that are close to the background color of the sign as much as possible to filter out the interference part in the background.

[0089] Specifically, in step S11, the panoramic image is converted from the RGB color space to the HSV color space.

[0090] In one embodiment, calculating the color space difference based on the RGB color space data of the panoramic image, and generating the HSV color space data according to the color space difference includes the following steps:

[0091] S111, standardizing the RGB color space data of the panoramic image to obtain standard color components;

[0092] S112, calculating the maximum and minimum values ​​of the standard color components, and solving the color space difference;

[0093] S113. Generate a hue component, a saturation component, and a brightness component based on the color space difference to obtain HSV color space data.

[0094] Specifically, the expression of standard color components is:

[0095]

[0096] Where R, G, and B are the red, green, and blue color components of the panoramic image, respectively; R′, G′, and B′ are the red, green, and blue color components of the standard color components, respectively;

[0097] The expression of color space difference is:

[0098]

[0099]

[0100] In the formula, I max is the maximum value among the standard color components, I min is the minimum value among the standard color components, and δ is the color space difference;

[0101] The expression of HSV color space data is:

[0102]

[0103]

[0104]

[0105] Wherein, H is the hue component of the HSV color space data, S is the saturation component of the HSV color space data, and V is the brightness component of the HSV color space data.

[0106] Specifically, in step S12, color segmentation is performed on the image in the HSV color space.

[0107] Specifically, traffic signs mainly include warning signs, prohibition signs, instruction signs and auxiliary signs, which are classified by color into blue background with white characters, yellow background with black characters and red background with white characters, and traffic sign color pixels are obtained. Traffic sign color pixels include blue, yellow, red and white. The combination of three components in the HSV color space is used to extract the traffic sign color pixels in the image, and the threshold adjustment function (in this embodiment, the trackbar function in OpenCV is used) is used to find the threshold range of the three components corresponding to the four colors:

[0108] Blue Threshold Range:

[0109] Yellow Threshold Range:

[0110] Red Threshold Range:

[0111] White threshold range:

[0112] Specifically, the three components are subjected to “AND” operation in the HSV color space for color segmentation. The segmentation result is as follows: Figure 2 As shown in the figure, (a) is the original RGB image, and (b) is the image after HSV color segmentation.

[0113] Specifically, in step S13, the image extracted by color segmentation contains a lot of noise, and median filtering is used to remove the redundant noise.

[0114] In one embodiment, based on the point cloud data in the laser radar coordinate system, coordinate transformation is performed using initial transformation parameters, and the exterior orientation elements of each panoramic image are combined to generate a corresponding panoramic depth image, and a target point cloud collection in the area of ​​interest of the traffic sign is extracted. The coordinate position of the traffic sign in the panoramic image is determined using a quantitative feature evaluation rule, including the following steps:

[0115] S21, performing coordinate transformation using initial transformation parameters according to the point cloud data in the laser radar coordinate system to obtain the point cloud data in the panoramic camera coordinate system, and solving the first characteristic angle and the second characteristic angle of the line connecting the spatial point and the camera center, combining the exterior orientation elements of each panoramic image, and generating a corresponding panoramic depth image;

[0116] S22, based on the panoramic depth image corresponding to each panoramic image, extracting the point cloud data falling into the area of ​​interest of the traffic sign, and establishing a target point cloud collection;

[0117] S23. Use quantitative feature evaluation rules to evaluate the matching degree of data points in the target point cloud collection, and determine the coordinate position of the traffic sign in the panoramic image based on the matching degree evaluation results of the data points, wherein the quantitative feature evaluation rules include height feature evaluation, shape feature evaluation and continuity feature evaluation.

[0118] It should be noted that the present invention uses HSV color segmentation to achieve the preliminary positioning of the region of interest containing the traffic sign. However, in natural scenes, there are inevitably some areas whose background colors are similar to the color gamut of the road traffic sign. This makes it difficult to eliminate the disturbed block areas in the results of HSV color segmentation, and thus the wrong region of interest cannot be removed.

[0119] Specifically, in order to address the problem that erroneous regions of interest cannot be removed, the present invention innovatively proposes in step S2 to utilize the high-precision three-dimensional spatial information characteristics of the collected point cloud data, deeply mine the key elements such as position coordinates, depth values, and rich geometric attributes contained therein, and perform feature matching and spatial positioning analysis on the regions of interest in the panoramic image, thereby filtering out erroneous regions of interest and achieving accurate positioning of signboards in the panoramic image.

[0120] In one embodiment, based on the point cloud data in the laser radar coordinate system, coordinate transformation is performed using the initial transformation parameters to obtain the point cloud data in the panoramic camera coordinate system, and the first characteristic angle and the second characteristic angle of the line connecting the space point and the camera center are solved, and the exterior orientation elements of each panoramic image are combined to generate a corresponding panoramic depth image, including the following steps:

[0121] S211, solving initial conversion parameters according to the point cloud data in the laser radar coordinate system, and using the initial conversion parameters to perform coordinate conversion to obtain point cloud data in the panoramic camera coordinate system, wherein the initial conversion parameters include a rotation matrix and a translation vector;

[0122] S212, based on the point cloud data in the panoramic camera coordinate system, obtaining the point cloud data in the spherical coordinate system by calculating the first characteristic angle and the second characteristic angle of the line connecting the spatial point and the camera center;

[0123] S213, determining the point cloud data in the pixel coordinate system based on the point cloud data in the spherical coordinate system and combining the exterior orientation elements of each panoramic image, and generating a corresponding panoramic depth image, wherein the exterior orientation elements of each panoramic image include the width and height of each panoramic image.

[0124] Specifically, in step S21, the point cloud is converted into a depth image corresponding to the panoramic image. The coordinates of the point cloud are converted using the exterior orientation elements of the panoramic image so that the points in the laser point cloud data correspond to the pixels in the image one by one. This process involves three coordinate systems: the world coordinate system (X w , Y w , Z w ), panoramic camera coordinate system (X c , Y c , Z c ) and the image plane coordinate system (u, v).

[0125] Specifically, in step S211, first, the point cloud data is converted from the UTM projection coordinates in the world coordinate system (WGS-84 coordinate system is used in this embodiment) to the coordinates in the local coordinate system in the POS system (the lidar coordinate system in this embodiment); then, through the initial conversion parameters R and T, the point cloud is converted from the lidar coordinate system to the panoramic camera coordinate system, so that the point cloud and the panoramic image are in the same reference system.

[0126] Specifically, the expression of point cloud data in the panoramic camera coordinate system is:

[0127]

[0128]

[0129]

[0130] In the formula, X c , Y c , Z c They are the point cloud data of the X-axis, Y-axis, and Z-axis in the panoramic camera coordinate system, R is the rotation matrix between the laser radar coordinate system and the panoramic camera coordinate system, X w , Y w , Z w are the point cloud data of the X-axis, Y-axis, and Z-axis in the laser radar coordinate system, T is the translation vector between the laser radar coordinate system and the panoramic camera coordinate system, and T X , T Y , T Z are the translation amounts in the X-axis, Y-axis, and Z-axis directions, θ X ,θ Y ,θ Z They are the rotation angles around the X-axis, Y-axis, and Z-axis respectively.

[0131] Specifically, in step S212, the point cloud is converted from the panoramic camera coordinate system to a spherical coordinate system with the camera position as the sphere center.

[0132] Specifically, the point cloud data in the spherical coordinate system includes a first characteristic angle and a second characteristic angle, and the expressions of the first characteristic angle and the second characteristic angle are:

[0133]

[0134] In the formula, is the angle between the line connecting the space point and the camera center and the Z axis, and θ is the angle between the projection of the line connecting the space point and the camera center on the XY plane and the X axis.

[0135] Specifically, in step S213, finally, the point cloud is transferred from the spherical coordinate system to the pixel coordinate system (image plane coordinate system) where the panoramic image is located, thereby completing the conversion of the laser point cloud from the three-dimensional space to the corresponding two-dimensional panoramic image plane.

[0136] Specifically, the expression of point cloud data in the pixel coordinate system is:

[0137]

[0138] Where u and v are the horizontal pixel coordinates and vertical pixel coordinates in the panoramic depth image, respectively, and H and W are the height and width of the panoramic image, respectively.

[0139] Specifically, the point cloud is mapped from the three-dimensional space to the two-dimensional space according to steps S211 to S213, and finally a panoramic depth image corresponding to the panoramic image is obtained, such as Figure 3 As shown, (a) is the original image of the panoramic image, and (b) is the corresponding panoramic depth image.

[0140] In one embodiment, based on the panoramic depth image corresponding to each panoramic image, point cloud data falling within the region of interest of the traffic sign is extracted to establish a target point cloud collection.

[0141] Specifically, in step S22, the point cloud points falling into each ROI are extracted respectively using the depth image as a medium. Each pixel point of the depth image is assigned a depth value, which is related to the three-dimensional coordinates in the point cloud data. Each point in the depth image represents a LiDAR point in three-dimensional space and corresponds to a pixel point in the panoramic image.

[0142] In one embodiment, quantitative feature evaluation rules are used to evaluate the matching degree of data points in the target point cloud collection, and based on the matching degree evaluation results of the data points, the coordinate position of the traffic sign in the panoramic image is determined, wherein the quantitative feature evaluation rules include height feature evaluation, shape feature evaluation and continuity feature evaluation.

[0143] Specifically, in step S23, the key features of the target are quantified and the feature matching degree is evaluated. The key features of the signboard include height, shape, continuity and mutation.

[0144] In one embodiment, the height feature evaluation includes calculating the height of the target area based on the panoramic image, and excluding the traffic sign interest area whose maximum value on the Y axis is greater than the height of the target area;

[0145] Shape feature assessment includes calculating the center point coordinates of all points in the traffic sign's area of ​​interest, the length of the line between the corner points, and the distance from the center point to any boundary point based on the target point cloud collection, and assessing whether the target area meets the shape features of circular, rectangular, and equilateral triangle signs;

[0146] The continuity feature evaluation includes plane fitting based on the target point cloud collection through the RANSAC algorithm, calculating the distance from the point to the plane and the plane equation coefficient, and evaluating the plane continuity and mutation of the target area.

[0147] Specifically, the target area is the correct area of ​​interest, that is, the area where the sign is believed to exist, and the relationship between this area and the sign area of ​​interest (ROI) in step S1 is a contained relationship; the purpose of setting three conditions as quantitative feature evaluation rules in the present invention is to filter out the erroneous areas of interest in the ROI obtained in step S1 through constraints such as height, shape and continuity, and obtain the correct area of ​​interest, that is, the target area.

[0148] Specifically, in the quantitative feature evaluation rule, the height feature evaluation is used as condition 1, including: the installation height of the road sign is not lower than the set height (set to 2 meters in this embodiment). Therefore, after reasonably adjusting the installation position and angle of the camera, the area of ​​interest of the sign is set to the upper 2 / 3 area of ​​the image. The expression of the target area height is:

[0149]

[0150] Where X and Y are the X-axis coordinate and Y-axis coordinate of the upper left corner of the target area, respectively; w and h are the width and height of the target area, respectively; w in With h in are the width and height of the panoramic image respectively;

[0151] Specifically, according to condition 1, if the ROI area is the maximum value y on the Y axis max >h, it is considered as an erroneous region of interest and is no longer considered.

[0152] Specifically, in the quantitative feature evaluation rule, shape feature evaluation is used as condition 2, including: the shapes of signboards are mainly rectangles, circles and equilateral triangles, the shape features of signboards are used for quantification, and the three-dimensional point cloud spatial information is used to filter out the mismatched areas. Let the target point cloud collection belonging to the ROI in the point cloud be P={p i =(x i ,y i ,z i )}, the center point C=(x c ,y c ,z c )}. By calculating the maximum x coordinate max and the minimum value x min , and the maximum value of the y coordinate y max and the minimum value y min , we get the length of the region l=x max -x min , width w = y max -y min By calculating the length of the line between the corner points in the ROI, we get two sets of opposite side lengths a and b. The points with the longest distance in the horizontal direction and the longest distance in the vertical direction are combined as corner points.

[0153] Specifically, the expression of the center point coordinates of all points in the area of ​​interest of the traffic sign is:

[0154]

[0155] In the formula, x c ,y c 、zc are the X-axis coordinate, Y-axis coordinate and Z-axis coordinate of the center point respectively, n is the number of all points in the area of ​​interest of the traffic sign, x i ,y i 、z i are the X-axis coordinate, Y-axis coordinate, and Z-axis coordinate of the i-th point in the target point cloud collection.

[0156] Specifically, according to condition 2:

[0157] ①If , and the distance from the center point to any boundary point , it is considered to be a circular sign area; is the allowable error value;

[0158] ②If , calculate the distance d from the center point to the boundary in four directions (horizontally and vertically) x and d y .like , ,and , it is considered to be a rectangular sign;

[0159] ③If and ,but , it is considered to be an equilateral triangle sign;

[0160] If the shape feature evaluation results do not meet the above ①, ②, ③, it is considered to be an incorrect region of interest and will not be considered again.

[0161] Specifically, in the quantitative feature evaluation rules, continuity feature evaluation is used as condition 3, including: utilizing the characteristics of the plane continuity of the signboard, using the RANSAC algorithm and performing plane fitting to filter out discontinuous and mutational point cloud data in the ROI.

[0162] Specifically, first, obtain point cloud data from ROI, and set the target point cloud collection as P={p i =(x i ,y i ,z i )}, randomly select three points p1, p2, and p3 from the target point cloud collection P, and use these three points to determine the parameters of the plane equation ax+by+cz+d=0; where a, b, and c are the normal vector components of the plane equation, and the three parameters together constitute the normal vector of the fitted plane =(a,b,c); The values ​​of a, b, and c vary with the inclination angle of the signboard. For example, the signboard facing the collector is perpendicular to the Z axis. In the plane equation, a=0.026, close to 0, means that the plane is almost parallel to the X axis; b=0.058, close to 0, means that the plane is almost parallel to the Y axis; c=0.998, close to 1, means that the plane is basically perpendicular to the Z axis; the normal vector of the fitting plane is Determines the "direction" of the plane in space; d is a constant term (in this example, d=-2.153), which represents the distance from the plane to the origin, and it determines the "position" of the plane in space. Then, for each point p in the target point cloud collection, i , calculate the distance from the point to the plane ; if d i If the value is less than the set inlier threshold t (in this embodiment, the inlier threshold t=0.05m), the point is recorded as an inlier. Finally, after multiple iterations, the plane model with the largest number of inliers is selected as the optimal plane.

[0163] Specifically, based on the optimal plane obtained by the above plane fitting, check its plane equation coefficient c. If it satisfies , where ε is a small threshold, the plane is considered to be the plane where the sign is located. The inner points in the plane are points with continuity. The points far from the plane are removed, that is, points with mutation.

[0164] Specifically, according to the three conditions of the quantitative feature evaluation rule, points that meet the three conditions are considered to have a high degree of feature matching, and the points with high matching degrees are retained, while the points with low matching degrees are removed. In this way, the coordinate position of the sign in the panoramic image is determined, and a bounding box surrounding the sign is generated based on the coordinates to achieve accurate positioning of the traffic sign area.

[0165] In one embodiment, a traffic sign sample library is established based on the collected panoramic image data, and a fusion feature vector is extracted by a local binary pattern algorithm and a directional gradient histogram algorithm to construct a traffic sign classification model; the traffic sign classification model is trained using the traffic sign sample library, the coordinate position of the traffic sign in the panoramic image is input, and the category information of the traffic sign is output, including the following steps:

[0166] S31. Based on the collected panoramic image data, a sign area is selected as a positive sample, and a non-sign area is selected as a negative sample to obtain a traffic sign sample library;

[0167] S32. Extract texture feature vectors using a local binary pattern algorithm, extract shape feature vectors using a directional gradient histogram algorithm, and use a principal component analysis method to reduce the dimension of the shape feature vectors to obtain a fused feature vector and establish a feature extraction network; based on the feature extraction network and combined with a one-to-many classification strategy, construct a traffic sign classification model;

[0168] S33. Train a traffic sign classification model using a traffic sign sample library, and based on the trained traffic sign classification model, input the coordinate position of the traffic sign in the panoramic image, and output the category information of the traffic sign.

[0169] Specifically, after determining the precise position of the sign in the panoramic image, the next step S3 is to identify the specific category of the sign through machine learning. First, through the collected panoramic image data, the sign area is manually cropped from the traffic sign area to make a positive sample, and an image of the same size is cropped from the non-traffic sign area as a negative sample. The positive samples contain three categories, namely, instruction signs, prohibition signs, and warning signs, which constitute the traffic sign sample library. Secondly, a traffic sign classification model is constructed, and the classification model consists of a feature extraction network and a classifier. Among them, in the feature extraction network, a feature extraction method that fuses texture information and shape information is proposed to achieve multi-feature fusion and complementarity, thereby improving recognition accuracy. The fusion strategy of the two features is as follows: Figure 4 shown.

[0170] Specifically, in this embodiment, the classifier uses an SVM classifier to achieve classification and recognition of traffic signs, and adopts a "one-to-many" strategy, that is, each type of sample is regarded as one category, and the remaining samples are classified as another category. In this way, four classifiers can be constructed for four types of samples, and the unknown samples are classified as the category with the largest classification function value during classification. Then, the constructed traffic sign classification model is trained using the prepared traffic sign sample library, and after multiple iterative training, a trained traffic sign classification model is obtained. Next, the trained classification model is used to identify the sign area located in the panoramic image to obtain specific category information. Finally, the category information and location information of the sign are combined to achieve accurate recognition of traffic signs.

[0171] Specifically, the feature extraction method for fusing texture information and shape information proposed in the present invention includes: first, the local binary pattern (LBP) feature vector that can describe the internal texture information of the sign image is calculated by the LBP algorithm. The LBP feature vector is a one-dimensional vector, and its dimension depends on the division method of the LBP feature map and the range of the LBP value. The LBP feature map is divided into 16 sub-regions, and the LBP value range is 0~255, then a one-dimensional LBP feature vector with a dimension of 4096 is obtained. Then, the HOG algorithm is used to calculate the directional gradient histogram (HOG) feature vector that can express the shape information of the sign. The HOG algorithm can convert a 3-channel color image into a one-dimensional feature vector of a certain length. Then, the PCA method is used to reduce the data dimension of the HOG feature vector. The feature vector F is reduced to f by the PCA algorithm as follows:

[0172] f=W T F

[0173] Where W is the mapping transformation matrix.

[0174] Specifically, the LBP feature vector and the PCA-HOG feature vector obtained after dimensionality reduction are concatenated in the feature dimension by a simple head-to-tail connection to obtain the fused feature vector PCA-HOG+LBP.

[0175] Specifically, each element of the PCA-HOG+LBP feature vector is mapped to the interval [0,1] using min-max normalization, and then input into the classifier for category judgment.

[0176] Specifically, the classification model constructed by using the prepared traffic sign samples is trained, and after multiple iterations, a trained SVM classifier is obtained. The fused feature vector is input into the SVM classifier, and for each category k, its decision function value f is calculated respectively. k (x), k=1,2,3,4 correspond to the direction sign, prohibition sign, warning sign and negative sample respectively. For each input feature vector x, four decision function values ​​f1(x), f2(x), f3(x), f4(x) are calculated, the values ​​are compared, and the sign in the area is assigned to the category with the largest decision value. The expression of the decision function is:

[0177] f k (x) = ω k T x+b k

[0178] Where x=(x1,x2,x3,…,x n ) is the fused feature vector, ω k =(ω1,ω2,ω3,…,ω n) is the weight vector corresponding to category k, b k is the bias term.

[0179] In order to facilitate understanding of the above technical solution of the present invention, the following is a specific description using 1000 panoramic images of traffic signs collected on a section of the west ring road in a certain urban area as an example:

[0180] During the data collection phase, a mobile measurement system equipped with a lidar and a panoramic camera was used to collect data on this road section. The collection equipment included a Ladybug5 panoramic camera and a VelodyneHDL-32E lidar, and the collection speed was maintained at around 40km / h. The collected data covered a road section of about 10 kilometers. In addition to the 1,000 panoramic images used for the experiment, it also included 1,000 panoramic images for verification and the corresponding point cloud data.

[0181] In the HSV color segmentation stage, the RGB image is first converted to the HSV space. For a panoramic image with a resolution of 8000×4000, its RGB value range is [0,255]. After normalization, the R', G', and B' values ​​are in the range of [0,1]. Then the maximum value I is calculated. max and minimum value I min , and the color space difference δ is obtained. Based on experimental statistics, the HSV threshold ranges of blue, yellow, red and white are set for color segmentation. The H value range of blue is [100,124], the S value range is [43,255], and the V value range is [46,255]; the H value range of yellow is [26,34], the S value range is [43,255], and the V value range is [46,255]; the H value range of red is [0,10] and [156,180], the S value range is [43,255], and the V value range is [46,255]; the H value range of white is [0,180], the S value range is [0,30], and the V value range is [221,255].

[0182] In the point cloud processing stage, the initial transformation parameters obtained by calibration include the rotation angle θ X =0.02°, θ Y =0.15°, θ Z=89.95°, translation vector T=(0.2m, 0.1m, 0.3m). The point cloud data is converted from the lidar coordinate system to the panoramic camera coordinate system, and then to the spherical coordinate system, and the characteristic angles φ and θ are calculated. Combined with the resolution parameters of the panoramic image (W=8000, H=4000), the corresponding depth image is generated. For each area of ​​interest, the point cloud data is extracted and the feature evaluation is performed, including setting the target area height threshold to 2 / 3 of the image height (about 2.67m), the aspect ratio threshold ε=0.1 of the rectangular sign, the side length error threshold δ=0.05m, and the RANSAC inlier threshold t=0.05m and the plane normal vector threshold ε=0.1 during plane fitting.

[0183] In the classification and recognition stage, 500 sign samples (including 200 instruction signs, 150 prohibition signs, and 150 warning signs) and 500 negative samples were manually cropped from the 2000 panoramic images collected and uniformly adjusted to 64×64 pixels. During feature extraction, the LBP feature dimension was 4096, the HOG feature dimension was 1000 after PCA dimension reduction, and the total dimension of the fusion feature was 5096. Linear kernel SVM was used for training, and the number of iterations was set to 1000. By comparing the performance of different feature extraction methods, it was found that in terms of running time, the average running time of PCA-HOG+LBP fusion features was slightly less than that of single HOG features; in contrast, the running time of using HOG+LBP features without dimensionality reduction increased a lot; the LBP feature data volume was small, so it had the least running time. In terms of recognition accuracy, the LBP algorithm features are simple and have the lowest recognition accuracy. Compared with the HOG+LBP fusion features and the reduced-dimensional PCA-HOG+LBP features, the fused features lag slightly behind in recognition accuracy, but the gap is not large, and the recognition speed is greatly optimized. In summary, the PCA-HOG+LBP fusion strategy achieves the best balance between recognition accuracy and speed, indicating that the feature extraction method for fusing texture information and shape information proposed in the present invention is effective.

[0184] The experimental results show that the traffic sign positioning method based on color features and depth image features proposed in the present invention has significant advantages. Through the organic combination of HSV color space segmentation and point cloud depth image features, the problem that traditional methods that rely solely on color features are easily interfered by similar backgrounds is effectively solved. On the 1,000 test images used for verification, this method achieved a sign detection accuracy of 95.3% and a positioning accuracy better than 0.5m. On this basis, the PCA-HOG+LBP feature fusion strategy was used for classification and recognition, which not only achieved a classification accuracy of 93.8%, but also controlled the average processing time to 0.8s / frame. The experimental results fully verified the effectiveness of the technical solution of the present invention that combines color features with depth image features. This method not only ensures the accuracy of traffic sign positioning, but also meets the requirements for processing efficiency in practical applications.

[0185] In summary, with the help of the above technical solution of the present invention, by decomposing positioning and category judgment into two independent steps, the traffic signs are accurately positioned using color features and depth information features. On the basis of accurate positioning, the classification model is used to distinguish the category of the signboard in the positioning area, thereby realizing high-precision recognition of traffic signs in complex traffic scenes. The present invention utilizes the distinct color characteristics of traffic signs in panoramic images and adopts HSV color segmentation to realize preliminary positioning of the signboard area. The method is simple in principle, small in calculation amount, and easy to implement. The present invention proposes a feature extraction method that integrates texture information and shape information. The HOG feature is difficult to present the overall texture of the traffic sign and is prone to errors when slightly occluded. The LBP feature is good at describing texture and is not afraid of slight deformation and occlusion. The present invention combines the two, complements each other's advantages, significantly improves the recognition accuracy, and enhances the performance and stability of the system in complex traffic scenes. Aiming at the problem that the extraction of traffic signs based on machine learning algorithms in road images is not efficient and accurate enough, the present invention utilizes the high-precision three-dimensional spatial information characteristics of point cloud data to perform feature matching and spatial positioning analysis on traffic signs in panoramic images, providing complete data support for accurately extracting corresponding target positions in images.

[0186] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A traffic sign positioning method based on color features and depth image features, characterized in that: The traffic sign positioning method based on color features and depth image features comprises the following steps: S1. Based on the RGB color space data of the panoramic image, HSV color space data including hue component, saturation component and brightness component is constructed, and the threshold range of each component is determined. By performing color segmentation processing on each component of the HSV color space data, the region of interest of the traffic sign is obtained; S2. Based on the point cloud data in the laser radar coordinate system, the initial transformation parameters are used to perform coordinate transformation, and the corresponding panoramic depth image is generated by combining the exterior orientation elements of each panoramic image. The target point cloud collection in the area of ​​interest of the traffic sign is extracted, and the coordinate position of the traffic sign in the panoramic image is determined by using quantitative feature evaluation rules; S3. Based on the collected panoramic image data, a traffic sign sample library is established, and the fused feature vector is extracted through the local binary pattern algorithm and the directional gradient histogram algorithm to construct a traffic sign classification model; the traffic sign classification model is trained using the traffic sign sample library, the coordinate position of the traffic sign in the panoramic image is input, and the category information of the traffic sign is output.

2. The method for locating traffic signs based on color features and depth image features according to claim 1, characterized in that: The method of constructing HSV color space data including hue component, saturation component and brightness component according to the RGB color space data of the panoramic image, determining the threshold range of each component, and obtaining the traffic sign interest area by performing color segmentation processing on each component of the HSV color space data includes the following steps: S11, calculating a color space difference based on the RGB color space data of the panoramic image, and generating HSV color space data according to the color space difference, wherein the HSV color space data includes a hue component, a saturation component, and a brightness component; S12, extracting the color pixels of the traffic sign in the panoramic image according to the hue component, the saturation component and the brightness component, and determining the threshold range of each component by using the threshold adjustment function, performing color segmentation processing on each component of the HSV color space data by AND operation, and obtaining a color segmentation image; S13. Use median filtering technology to remove noise in the color segmentation image and obtain the region of interest of the traffic sign.

3. The method for locating traffic signs based on color features and depth image features according to claim 2, characterized in that: The method of calculating the color space difference based on the RGB color space data of the panoramic image and generating the HSV color space data according to the color space difference comprises the following steps: S111, standardizing the RGB color space data of the panoramic image to obtain standard color components; S112, calculating the maximum and minimum values ​​of the standard color components, and solving the color space difference; S113. Generate a hue component, a saturation component, and a brightness component based on the color space difference to obtain HSV color space data.

4. The method for locating traffic signs based on color features and depth image features according to claim 3, characterized in that: The expression of the standard color component is: ； Where R, G, and B are the red, green, and blue color components of the panoramic image, respectively; R′, G′, and B′ are the red, green, and blue color components of the standard color components, respectively; The expression of the color space difference is: ； ； In the formula, I max is the maximum value among the standard color components, I min is the minimum value among the standard color components, and δ is the color space difference; The expression of the HSV color space data is: ； ； ； Wherein, H is the hue component of the HSV color space data, S is the saturation component of the HSV color space data, and V is the brightness component of the HSV color space data.

5. The method for locating traffic signs based on color features and depth image features according to claim 2, characterized in that: The method comprises the following steps: performing coordinate transformation based on the point cloud data in the laser radar coordinate system using the initial transformation parameters, combining the exterior orientation elements of each panoramic image, generating a corresponding panoramic depth image, extracting the target point cloud collection in the area of ​​interest of the traffic sign, and determining the coordinate position of the traffic sign in the panoramic image using the quantitative feature evaluation rule: S21, performing coordinate transformation using initial transformation parameters according to the point cloud data in the laser radar coordinate system to obtain the point cloud data in the panoramic camera coordinate system, and solving the first characteristic angle and the second characteristic angle of the line connecting the spatial point and the camera center, combining the exterior orientation elements of each panoramic image, and generating a corresponding panoramic depth image; S22, based on the panoramic depth image corresponding to each panoramic image, extracting the point cloud data falling into the area of ​​interest of the traffic sign, and establishing a target point cloud collection; S23. Use quantitative feature evaluation rules to evaluate the matching degree of data points in the target point cloud collection, and determine the coordinate position of the traffic sign in the panoramic image based on the matching degree evaluation results of the data points, wherein the quantitative feature evaluation rules include height feature evaluation, shape feature evaluation and continuity feature evaluation.

6. The method for locating traffic signs based on color features and depth image features according to claim 5, characterized in that: The method of performing coordinate transformation using initial transformation parameters according to the point cloud data in the laser radar coordinate system to obtain the point cloud data in the panoramic camera coordinate system, solving the first characteristic angle and the second characteristic angle of the line connecting the space point and the camera center, and combining the exterior orientation elements of each panoramic image to generate a corresponding panoramic depth image includes the following steps: S211, solving initial conversion parameters according to the point cloud data in the laser radar coordinate system, and using the initial conversion parameters to perform coordinate conversion to obtain point cloud data in the panoramic camera coordinate system, wherein the initial conversion parameters include a rotation matrix and a translation vector; S212, based on the point cloud data in the panoramic camera coordinate system, obtaining the point cloud data in the spherical coordinate system by calculating the first characteristic angle and the second characteristic angle of the line connecting the spatial point and the camera center; S213. Determine the point cloud data in the pixel coordinate system based on the point cloud data in the spherical coordinate system and in combination with the exterior orientation elements of each panoramic image, and generate a corresponding panoramic depth image, wherein the exterior orientation elements of each panoramic image include the width and height of each panoramic image.

7. The method for locating traffic signs based on color features and depth image features according to claim 6, characterized in that: The expression of the point cloud data in the panoramic camera coordinate system is: ； Where, X c , Y c , Z c They are the point cloud data of the X-axis, Y-axis, and Z-axis in the panoramic camera coordinate system, R is the rotation matrix between the laser radar coordinate system and the panoramic camera coordinate system, X w , Y w , Z w are the point cloud data of the X-axis, Y-axis, and Z-axis in the laser radar coordinate system, and T is the translation vector between the laser radar coordinate system and the panoramic camera coordinate system; The point cloud data in the spherical coordinate system includes a first characteristic angle and a second characteristic angle, and the expressions of the first characteristic angle and the second characteristic angle are: ； In the formula, is the angle between the line connecting the space point and the camera center and the Z axis, and θ is the angle between the projection of the line connecting the space point and the camera center on the XY plane and the X axis; The expression of the point cloud data in the pixel coordinate system is: ； Where u and v are the horizontal pixel coordinates and vertical pixel coordinates in the panoramic depth image, respectively, and H and W are the height and width of the panoramic image, respectively.

8. The method for locating traffic signs based on color features and depth image features according to claim 5, characterized in that: The height feature evaluation includes calculating the height of the target area based on the panoramic image, and excluding the traffic sign interest area whose maximum value on the Y axis is greater than the height of the target area; The shape feature evaluation includes calculating the center point coordinates of all points in the traffic sign area of ​​interest, the length of the line between the corner points, and the distance from the center point to any boundary point based on the target point cloud collection, and evaluating whether the target area meets the shape features of the traffic sign of a circle, rectangle, or equilateral triangle; The continuity feature evaluation includes performing plane fitting based on the target point cloud collection through the RANSAC algorithm, calculating the distance from the point to the plane and the plane equation coefficient, and evaluating the plane continuity and mutation of the target area.

9. The method for locating traffic signs based on color features and depth image features according to claim 8, characterized in that: The expression of the target area height is: ； Where X and Y are the X-axis coordinate and Y-axis coordinate of the upper left corner of the target area, respectively; w and h are the width and height of the target area, respectively; w in With h in are the width and height of the panoramic image respectively; The expression of the center point coordinates of all points in the area of ​​interest of the traffic sign is: ； In the formula, x c ,y c 、z c are the X-axis coordinate, Y-axis coordinate and Z-axis coordinate of the center point respectively, n is the number of all points in the area of ​​interest of the traffic sign, x i ,y i 、z i are the X-axis coordinate, Y-axis coordinate, and Z-axis coordinate of the i-th point in the target point cloud collection.

10. The method for locating traffic signs based on color features and depth image features according to claim 1, characterized in that: The method of establishing a traffic sign sample library based on the collected panoramic image data, extracting fusion feature vectors through a local binary pattern algorithm and a directional gradient histogram algorithm, and constructing a traffic sign classification model; using the traffic sign sample library to train the traffic sign classification model, inputting the coordinate position of the traffic sign in the panoramic image, and outputting the category information of the traffic sign includes the following steps: S31. Based on the collected panoramic image data, a sign area is selected as a positive sample, and a non-sign area is selected as a negative sample to obtain a traffic sign sample library; S32. Extract texture feature vectors using a local binary pattern algorithm, extract shape feature vectors using a directional gradient histogram algorithm, and use a principal component analysis method to reduce the dimension of the shape feature vectors to obtain a fused feature vector and establish a feature extraction network; based on the feature extraction network and combined with a one-to-many classification strategy, construct a traffic sign classification model; S33. Train a traffic sign classification model using a traffic sign sample library, and based on the trained traffic sign classification model, input the coordinate position of the traffic sign in the panoramic image, and output the category information of the traffic sign.

Images