Road traffic sign recognition system for automatic driving

Abstract

The invention discloses a road traffic sign recognition system for automatic driving, which belongs to the technical field of image processing and comprises a sensing and acquisition module, a preprocessing module, a traffic sign detection module, a traffic sign recognition module and a vehicle control module which are connected in sequence. Wherein the sensing and collecting module is used for collecting sensor data and environment data; the preprocessing module is used for preprocessing the collected sensor data and the environment data to obtain a fused image; the traffic sign detection module is used for performing target detection on the fused image based on a region suggestion network to obtain a traffic sign region; the traffic sign recognition module recognizes the traffic sign area to obtain traffic sign information; the vehicle control module is used for providing road feedback information for a vehicle driver based on the traffic sign information.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a road traffic sign recognition system for automatic driving. Background Art

[0002] Traffic sign recognition technology in the field of autonomous driving is one of the key technologies to ensure safe driving of vehicles. At present, methods based on deep learning, especially convolutional neural networks (CNNs), have become the mainstream technology for traffic sign recognition. These models can automatically extract image features and train on large-scale annotated datasets. At the same time, attention mechanisms are introduced to enhance the model's attention to key areas in the image and improve recognition accuracy.

[0003] Although deep learning models excel in feature extraction, they still have shortcomings in global semantic logic analysis. These models usually focus on local features and lack the ability to understand and reason about the logical relationship between traffic signs. For example, a "No Entry" sign may be recognized, but the model may not be able to infer that this means that all roads entering the area are prohibited. At the same time, in fast-moving vehicles or in complex dynamic environments, the rapid appearance and disappearance of signs may cause recognition failures; occlusion by dynamic objects (such as other vehicles and pedestrians) may interfere with the detection and recognition of signs; it can be seen that traffic sign recognition systems need to be able to accurately identify static signs in dynamic scenes, but existing technologies are not sensitive enough to dynamic interaction processes.

[0004] Therefore, although traffic sign recognition technology in the field of autonomous driving has made significant progress, in order to achieve a safer and more reliable autonomous driving system, it is still necessary to solve problems such as insufficient global semantic logic analysis and sensitivity to dynamic interactive recognition processes. Future research needs to focus on improving the model's contextual understanding ability, dynamic scene adaptability, and environmental robustness, while considering computational efficiency and real-time performance to meet the actual application requirements of autonomous driving systems. Therefore, based on the deficiencies of the prior art, the present invention proposes a road traffic sign recognition system for autonomous driving. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a road traffic sign recognition system for autonomous driving to solve the above problems existing in the prior art.

[0006] To achieve the above object, the present invention provides a road traffic sign recognition system for automatic driving, comprising:

[0007] A perception and acquisition module, a preprocessing module, a traffic sign detection module, a traffic sign recognition module and a vehicle control module connected in sequence;

[0008] Wherein, the perception and acquisition module is used to collect sensor data and environmental data;

[0009] The preprocessing module is used to preprocess the collected sensor data and environmental data respectively to obtain a fused image;

[0010] The traffic sign detection module is used to perform target detection on the fused image based on a region proposal network to obtain a traffic sign region;

[0011] The traffic sign recognition module recognizes the traffic sign area to obtain traffic sign information;

[0012] The vehicle control module is used to provide road feedback information to the vehicle driver based on the traffic sign information.

[0013] Optionally, the sensing and acquisition module includes a sensor integration unit and an image acquisition unit;

[0014] Wherein, the sensor integration unit is used to obtain sensor data by capturing vehicle surrounding environment data based on the vehicle-mounted camera, radar, laser radar and GPS;

[0015] The image acquisition unit is used to acquire the environment image of the vehicle-mounted camera.

[0016] Optionally, the preprocessing module includes a sensor preprocessing unit, an image preprocessing unit and an image fusion unit;

[0017] Wherein, the sensor preprocessing unit is used to preprocess the sensor data to obtain preprocessed sensor data;

[0018] The image preprocessing unit is used to preprocess the environment image to obtain a preprocessed image;

[0019] The image fusion unit is used to fuse the preprocessed sensor data and the preprocessed image to obtain a fused image based on a method for estimating the coordinates of the vehicle-mounted camera based on image feature points.

[0020] Optionally, the sensor preprocessing unit includes a first preprocessing subunit and a sensor data fusion subunit;

[0021] Wherein, the first preprocessing subunit is used to perform standardization and denoising on the sensor data based on filtering processing and data format standardization processing to obtain standard format data corresponding to different sensors;

[0022] The sensor data fusion subunit is used to fuse the standard format data corresponding to different sensors based on a feature-level fusion method to obtain pre-processed sensor data.

[0023] Optionally, the image preprocessing unit includes a dedistortion subunit and a format processing subunit;

[0024] Wherein, the dedistortion subunit is used to dedistort the environment image to obtain a dedistorted image based on camera calibration theory;

[0025] The format processing subunit is used to perform format unification processing on the dedistorted image to obtain a preprocessed image.

[0026] Optionally, the traffic sign recognition module includes: a feature sharing unit, a feature branching unit, a semantic segmentation unit and a multi-task learning unit;

[0027] Wherein, the feature sharing unit is used to obtain the general image features of the traffic sign area;

[0028] The feature branching unit is used to obtain global-local features based on the general features of the image;

[0029] The semantic segmentation unit is used to obtain semantic information based on the global-local features;

[0030] The multi-task learning unit is used to optimize the semantic information to obtain traffic sign classification results and traffic sign information.

[0031] Optionally, the feature sharing unit includes a convolution layer, an activation layer, a pooling layer, a normalization layer, a residual connection layer, a space segment convolution layer, a depth-separable convolution and a fully connected layer.

[0032] Optionally, the feature branch unit includes a global feature branch sub-unit, a local feature branch sub-unit and a motion state branch sub-unit;

[0033] Wherein, the global feature branch subunit is used to obtain the global features of the image;

[0034] The local feature branch subunit is used to obtain local features of the image;

[0035] The dynamic state branch subunit is used to obtain image motion features.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] The improved system disclosed in the present invention can more accurately understand the logical relationship between traffic signs and make safer and more reasonable driving decisions. At the same time, the system's adaptability to dynamic scenes is enhanced, reducing recognition errors caused by environmental changes and providing a smoother driving experience. In addition, the improvement in recognition accuracy will reduce misjudgments and missed judgments and optimize the decision-making process. The improvement in the system's generalization ability enables it to adapt to a wider range of environments and conditions, strictly abide by traffic rules, reduce violations, and support complex scene processing. With the improvement in performance, the need for manual intervention will decrease, and users' trust in autonomous driving technology will increase, which will help promote the commercialization and popularization of autonomous driving technology and provide a solid foundation for the safe operation and widespread application of autonomous driving vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] Embodiment 1

[0043] like Figure 1 As shown, this embodiment provides a road traffic sign recognition system for autonomous driving, including: a perception and acquisition module, a preprocessing module, a traffic sign detection module, a traffic sign recognition module and a vehicle control module connected in sequence.

[0044] As a preferred solution of this embodiment, the perception and acquisition module captures data of the vehicle's surrounding environment based on the on-board camera, radar, laser radar (LiDAR), GPS and other sensors of the sensor integration unit; then, the image acquisition unit obtains the environmental image in real time from the on-board camera; wherein, the vehicle is equipped with multiple on-board cameras, which can capture high-resolution images from different angles, including but not limited to front-view, side-view and rear-view cameras, which provide 360-degree visual coverage for the vehicle.

[0045] In addition to visual information, vehicles use sensors such as radar and laser radar (LiDAR) to detect the distance, speed and direction of surrounding objects. Radar sensors are particularly good at detecting metal objects, such as other vehicles, and can maintain good performance in adverse weather conditions. LiDAR creates an accurate three-dimensional map of the surrounding environment by emitting a laser beam and measuring the time it takes to reflect back, which is very effective for identifying non-metallic objects such as pedestrians, bicycles, traffic signs and lights. GPS provides the vehicle's precise location and direction of travel.

[0046] The collected images and sensor data will be transmitted to the preprocessing module, which preprocesses the image data and sensor data to obtain preprocessed images and preprocessed sensor data respectively.

[0047] As a preferred solution of this embodiment, the process of preprocessing the collected environment image and sensor data respectively based on the preprocessing module includes:

[0048] The sensor data is preprocessed based on the sensor preprocessing unit to obtain preprocessed sensor data, wherein the sensor preprocessing unit includes: a first preprocessing subunit and a sensor data fusion subunit. The specific implementation process includes: firstly, based on the first preprocessing subunit, the data collected by multiple sensors are filtered to obtain denoised data; the denoised data obtained based on different sensors are calibrated to obtain calibration data to ensure data synchronization and consistency between different sensors; the standard data is formatted to obtain standard format data; the information of the standard format data of different sensors is fused based on the sensor fusion subunit to form a comprehensive and reliable environmental model. Among them, the sensor fusion subunit extracts features from different standard format data based on feature-level fusion means, and obtains preprocessed sensor data based on the extracted data features.

[0049] The environmental image collected by the vehicle-mounted camera is preprocessed based on the image preprocessing unit to obtain a preprocessed image. Among them, the image preprocessing unit includes: a dedistortion subunit and a format processing subunit. The specific implementation process includes: dedistorting the environmental image based on the dedistortion subunit, and this embodiment calculates the intrinsic parameter matrix and distortion coefficient based on the camera calibration theory in the vehicle-mounted camera to perform dedistortion operation on the collected environmental data to obtain a dedistorted image; this embodiment saves the dedistorted image into a unified format based on the format processing subunit to obtain a preprocessed image.

[0050] This embodiment fuses the preprocessed sensor data and the preprocessed image based on the image fusion unit to obtain a fused image; the specific implementation process includes: extracting image features of the preprocessed image based on a corner detection method, estimating the movement of the vehicle camera based on feature points of the image features, and then performing post-processing on the image including enhancing image contrast and brightness to obtain a fused image.

[0051] As a preferred solution of this embodiment, the process of inputting the obtained fused image into the traffic sign detection module for traffic sign recognition in this embodiment includes: extracting features from the fused image based on the Region Proposal Network (RPN) to generate a fused feature map, and the fused feature map generates a candidate region based on a sliding window, wherein the RPN uses two parallel convolutional layers to predict whether the region contains a traffic sign (i.e., a classification task) and the bounding box offset of the region (i.e., a regression task). The classification convolutional layer outputs the probability that each sliding window is a background or a target, while the regression convolutional layer outputs an offset that adjusts the sliding to better match the actual traffic sign position. Subsequently, the overlapping candidate regions are merged based on the non-maximum suppression (NMS) algorithm to obtain a refined region, and the traffic sign region is obtained based on the refined region.

[0052] As a preferred solution of this embodiment, the traffic sign area is input into the traffic sign recognition module, and the traffic sign recognition module obtains traffic sign information by recognizing and processing the traffic sign area by constructing an improved multi-task learning module. The traffic sign recognition module includes: a feature sharing unit, a feature branching unit, a semantic segmentation unit, and a multi-task learning unit.

[0053] Among them, the feature sharing units include convolutional layers, activation layers, pooling layers, normalization layers, residual connection layers, empty segment convolution layers, depthwise separable convolution layers and fully connected layers.

[0054] The feature branch unit includes a global feature branch subunit, a local feature branch subunit, and a motion state branch subunit; the global feature branch subunit includes a global average pooling layer and a deep fully connected network; the local feature branch subunit includes a feature pyramid network, which has five convolutional layers, three maximum pooling layers, and three deconvolutional layers. The number of convolution kernels in all convolutional layers is C, and the kernel size is 3×3. In each maximum pooling layer, the kernel size is set to 2×2 and the padding is set to 1 to ensure that the input and output sizes are the same. In the three deconvolution layers, the kernel size is also set to 2×2 and the stride is set to 2. The output of the pyramid module is a weight map. Finally, the obtained weight map is multiplied with the input feature map to obtain the final weighted feature map; the motion state branch subunit includes several optical flow image convolutional layers, temporal convolutional layers, residual layers, bidirectional convolutional LSTM layers, and upsampling layers.

[0055] The semantic segmentation unit consists of a CBR layer, two fusion layers, and a cross-stage local network.

[0056] The multi-task learning unit consists of a fully connected layer, a recursive layer, an optimizer, and an output layer.

[0057] The specific implementation process includes: the traffic sign recognition module receives the traffic sign area through the input layer, and then obtains the common features in the image through the feature sharing unit. The process of obtaining the common features based on the feature sharing unit is: the traffic sign area is sequentially processed by the convolution layer, activation layer, pooling layer, normalization layer, and residual connection layer to obtain the first feature, the first feature is processed by the space segment convolution layer and the depthwise separable convolution layer to obtain the second feature and the third feature, the second feature and the third feature are unified in dimension through the convolution layer, and connected with the fully connected layer to obtain the common features of the image.

[0058] The general features of the image are input into the feature branch unit to obtain the global features of the image, the local features of the image and the motion features of the image. The specific implementation process includes: inputting the general features of the image into the global feature branch subunit, and sequentially passing through the global average pooling layer and the deep fully connected network to obtain the global features of the image; after the general features of the image are input into the local feature branch subunit, the fourth features are obtained through the convolution layer, and the fourth features are sequentially subjected to three feature convolutions to obtain the fifth features, wherein each feature convolution performs one pooling and deconvolution, and the fifth feature is again passed through the convolution layer to obtain the weight map, and the weight map and the input general feature map of the image are multiplied to obtain the weighted feature map, and the local features of the image are obtained based on the weighted feature map. The general features of the image are input into the motion state branch subunit through the convolution layer to obtain the basic features of the optical flow image, and the general features of the image are subjected to time convolution to extract the time dimension features, and then the basic features of the optical flow image and the time dimension features are connected through the residual layer to obtain the sixth feature, and the sixth feature is input into the bidirectional convolution LSTM layer to capture the dynamic features of the traffic signs in the time dimension to obtain the seventh feature, and predict the future optical flow changes; the seventh feature is upsampled to restore the spatial resolution of the feature map to obtain the image motion feature.

[0059] The image motion feature is input into the semantic segmentation unit, and the eighth feature is obtained through the CBS layer, where the CBS layer includes a convolution layer, a regularization layer and an activation function; the image local feature is input into the semantic segmentation unit, and the eighth feature and the input image local feature are fused to obtain the ninth feature; the image global feature is input into the semantic segmentation unit, and the ninth feature and the input image global feature are fused to obtain semantic information through the fusion layer and the cross-stage local network.

[0060] The semantic information is input into the multi-task learning unit, and the traffic sign classification results and traffic sign information are obtained through the fully connected layer, recursive layer, optimizer and output layer.

[0061] Among them, the loss function is defined for the optimizer, L global , L local , L motion They are the losses of global features, local features and motion state tasks respectively.

[0062] Ltotal =λ 1* L global +λ2*L local +λ3*L motion

[0063] Where, L total represents the total loss, λ1 represents the weight of the global feature loss, λ2 represents the weight of the local feature loss, and λ3 represents the weight of the motion state task loss.

[0064] As a preferred solution of this embodiment, after the traffic sign detection module recognizes the traffic sign information, the obtained traffic sign information is sent to the vehicle control module to provide voice broadcast service to the driver, and automatically start the vehicle braking mode in an emergency.

[0065] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A road traffic sign recognition system for autonomous driving, characterized in that: include: A perception and acquisition module, a preprocessing module, a traffic sign detection module, a traffic sign recognition module and a vehicle control module connected in sequence; Wherein, the perception and acquisition module is used to collect sensor data and environmental data; The preprocessing module is used to preprocess the collected sensor data and environmental data respectively to obtain a fused image; The traffic sign detection module is used to perform target detection on the fused image based on a region proposal network to obtain a traffic sign region; The traffic sign recognition module recognizes the traffic sign area to obtain traffic sign information; The vehicle control module is used to provide road feedback information to the vehicle driver based on the traffic sign information.

2. The road traffic sign recognition system for automatic driving according to claim 1, characterized in that: The sensing and acquisition module includes a sensor integration unit and an image acquisition unit; Wherein, the sensor integration unit is used to obtain sensor data by capturing vehicle surrounding environment data based on the vehicle-mounted camera, radar, laser radar and GPS; The image acquisition unit is used to acquire the environment image of the vehicle-mounted camera.

3. The road traffic sign recognition system for automatic driving according to claim 1, characterized in that: The preprocessing module includes a sensor preprocessing unit, an image preprocessing unit and an image fusion unit; Wherein, the sensor preprocessing unit is used to preprocess the sensor data to obtain preprocessed sensor data; The image preprocessing unit is used to preprocess the environment image to obtain a preprocessed image; The image fusion unit is used to fuse the preprocessed sensor data and the preprocessed image to obtain a fused image based on a method for estimating the coordinates of the vehicle-mounted camera based on image feature points.

4. The road traffic sign recognition system for automatic driving according to claim 3, characterized in that: The sensor preprocessing unit includes a first preprocessing subunit and a sensor data fusion subunit; Wherein, the first preprocessing subunit is used to perform standardization and denoising on the sensor data based on filtering processing and data format standardization processing to obtain standard format data corresponding to different sensors; The sensor data fusion subunit is used to fuse the standard format data corresponding to different sensors based on a feature-level fusion method to obtain pre-processed sensor data.

5. The road traffic sign recognition system for automatic driving according to claim 3, characterized in that: The image preprocessing unit includes a dedistortion subunit and a format processing subunit; Wherein, the dedistortion subunit is used to dedistort the environment image to obtain a dedistorted image based on camera calibration theory; The format processing subunit is used to perform format unification processing on the dedistorted image to obtain a preprocessed image.

6. The road traffic sign recognition system for automatic driving according to claim 1, characterized in that: The traffic sign recognition module includes: a feature sharing unit, a feature branching unit, a semantic segmentation unit and a multi-task learning unit; Wherein, the feature sharing unit is used to obtain the general image features of the traffic sign area; The feature branching unit is used to obtain global-local features based on the general features of the image; The semantic segmentation unit is used to obtain semantic information based on the global-local features; The multi-task learning unit is used to optimize the semantic information to obtain traffic sign classification results and traffic sign information.

7. The road traffic sign recognition system for automatic driving according to claim 6, characterized in that: The feature sharing unit includes a convolution layer, an activation layer, a pooling layer, a normalization layer, a residual connection layer, a space segment convolution layer, a depth-separable convolution and a fully connected layer.

8. The road traffic sign recognition system for automatic driving according to claim 6, characterized in that: The feature branch unit includes a global feature branch sub-unit, a local feature branch sub-unit and a motion state branch sub-unit; Wherein, the global feature branch subunit is used to obtain the global features of the image; The local feature branch subunit is used to obtain local features of the image; The dynamic state branch subunit is used to obtain image motion features.

Images