A traffic sign recognition method and system for active safety of commercial vehicles

Abstract

The application discloses a traffic sign recognition method and system for active safety of commercial vehicles, and specific steps of the recognition method are as follows: S1, capturing road panoramic image information through a vehicle-mounted camera device, and performing pretreatment operation on the road panoramic image information, and setting a threshold value through color transformation space to extract an image of a region of interest; S2, matching the image of the region of interest with a traffic sign template library picture, and positioning a sign image in the road panoramic image according to a matching result; S3, designing and building a neural network model, training the neural network model on a sampling traffic sign data set, and obtaining an optimal traffic sign recognition model; S4, merging a multi-branch structure in the optimal traffic sign recognition model, and obtaining a remodeling model; S5, performing classification recognition on the sign image in step S2, inputting the sign image into the remodeling model in step S4 after normalizing the sign image, and judging a traffic sign type according to an output result.

Description

Technical Field

[0001] This invention belongs to the field of active safety technology for commercial vehicles, and specifically relates to a traffic sign recognition method and system for active safety of commercial vehicles. Background Technology

[0002] With the development of intelligent driving technology and even autonomous driving technology, vehicle intelligence has become a key focus for automakers. Due to their inherent characteristics and role in economic production, commercial vehicles suffer more significant losses of life and property in traffic accidents than passenger vehicles. Therefore, improving the safety of commercial vehicles, especially through the application of active safety technologies, is of paramount value.

[0003] Traffic sign recognition is a crucial component of active safety technology. It provides vital information and data to active safety systems, and the efficient and accurate identification and judgment of traffic signs can assist drivers in decision-making, thereby achieving the goal of safe driving. Therefore, traffic sign recognition plays an indispensable role in the active safety of commercial vehicles.

[0004] However, existing traffic sign recognition methods have the following problems:

[0005] 1) The recognition accuracy is poor under the influence of outdoor lighting changes and complex weather scenes;

[0006] 2) The problem of poor real-time performance after complex algorithms are deployed to embedded devices. Summary of the Invention

[0007] To address the aforementioned technical problems in the existing technology, the first objective of this invention is to overcome the shortcomings and deficiencies of the existing technology and provide a fast and accurate traffic sign recognition method. This method can effectively improve the accuracy of traffic sign recognition under conditions such as changes in lighting, shadows, and rainy / foggy weather. At the same time, this method can achieve real-time sign detection on embedded devices. The second objective of this invention is to provide a traffic sign recognition system.

[0008] The technical solution adopted in this invention is:

[0009] A traffic sign recognition method for active safety of commercial vehicles, characterized in that the specific steps of the recognition method are as follows:

[0010] S1. Capture panoramic road image information through the vehicle-mounted camera device, preprocess the panoramic road image information, and extract the region of interest image by setting a threshold in the color transformation space.

[0011] S2. Match the above-mentioned region of interest image with the traffic sign template library image, and locate the sign image in the road panoramic image based on the matching result;

[0012] S3. Design and build a neural network model, train the neural network model on the sampled traffic sign dataset, and obtain the optimal traffic sign recognition model;

[0013] S4. Merge the multi-branch structure in the above optimal traffic sign recognition model to obtain the reshaped model;

[0014] S5. Classify and identify the sign images described in step S2. After normalizing the sign images, input them into the reshaping model in step S4. Determine the type of traffic sign based on the output results.

[0015] Furthermore, in step S1, obtaining the region of interest image specifically includes the following steps:

[0016] S11: The vehicle-mounted camera device acquires a road image with a width of width pixels and a height of height pixels;

[0017] S12: Retain the upper image of the captured road image at a height of h times, and the width and height of the cropped image are I... width I height It can be calculated using the following formulas (1) and (2):

[0018] I wide =width (1)

[0019] I height =h·height (2)

[0020] S13: Perform smoothing filtering on the upper image described in step S12. Assuming a window of size m×n is selected to filter the upper image, the calculation formula at the center pixel (x,y) of the filtering window is as follows (3):

[0021]

[0022] Where w(r,s) is the weighting function, r and s are the horizontal and vertical coordinate offsets of the current calculated pixel in the filtering window relative to the center pixel of the filtering window, respectively. Different weights are assigned to the weighting function according to the actual engineering needs. g(x,y) is the original image, and a and b are integers.

[0023] S14: Perform HSV spatial transformation on the image after filtering in step S13 above. Assuming that the red channel value is R, the green channel value is G, and the blue channel value is B at a certain point in the current image, the H, S, and V values ​​after HSV spatial transformation can be obtained through the following specific transformation steps:

[0024] S141: Extract the red channel value R, green channel value G, and blue channel value B from the RGB image;

[0025] S142: Normalize R, G, and B to obtain R', G', and B', as shown in equations (4), (5), and (6) below:

[0026] R'=R / 255 (4)

[0027] G' = G / 255 (5)

[0028] B' = B / 255 (6)

[0029] S143: Calculate the maximum channel value C max and minimum channel value C min The calculation formulas are shown in equations (7) and (8) below;

[0030] C max =max(R',G'B') (7)

[0031] C min =min(R',G'B') (8)

[0032] S144: The hue (H), saturation (S), and brightness (V) of an RBG format image converted to HSV space are calculated using the following formulas (9), (10), and (11):

[0033]

[0034]

[0035] V = C max (11)

[0036] S15, set the thresholds of H, S, and V in step S14 to obtain the mask, multiply the mask with the upper image in step S13 to obtain the region of interest image, and calculate it as shown in the following formula (12):

[0037]

[0038] Where B(x,y) represents the pixel value of the image of interest at coordinates (x,y), and O x,y I represents the pixel value at coordinates (x, y) in the original image. x,y This represents the pixel value of the mask at coordinates (x, y).

[0039] Furthermore, step S2 specifically includes the following steps:

[0040] S21: Create a traffic sign template library, which includes the following steps:

[0041] S211: For each type of mark to be detected, select a clear and identifiable mark image as a sample;

[0042] S212: Convert the selected logo image from RGB format to HSV format. The conversion process is completed according to the above steps S142, S143 and S144 so that subsequent processing can be performed in the HSV color space.

[0043] S213: In the HSV color space, based on a set threshold, an extraction operation is performed on the red, yellow, or blue regions to obtain the target color region;

[0044] S214: Obtain a template image from the extracted target color region, the template image representing the shape and color features of a specific type of logo;

[0045] S215: Combine the processed and extracted template images to form a template library containing various types of logos;

[0046] S22: Set a list of width and height scaling factors for scaling the template image. This list contains multiple sets of width and height scaling factors.

[0047] S23: Perform template matching on the aforementioned template library images and the aforementioned HSV segmentation images; for each set of width and height scaling factors, during the matching process, scale the template images according to the current scaling factors, and then match them with the HSV segmentation images;

[0048] S24: During the template matching process, the matching score corresponding to each set of coefficients is recorded. The score is calculated using the following formula (13):

[0049]

[0050] Where Score represents the matching score between the two, T' represents the template pixel value, and I' represents the target image pixel value. The value is 1 when the template and the sliding window match perfectly, and -1 when they do not match at all.

[0051] S25: Select the set of coefficients with the highest scores from all matching scores as the final scaling factor for the template;

[0052] S26: Scale the template height and width according to the aforementioned scaling factors, and calculate the scaled width and height of the template using formulas (14) and (15):

[0053] w new =w coef ·w old (14)

[0054] h new =h coef ·h old (15)

[0055] Among them, w new h new The width and height of the template after scaling are respectively, w old h old These represent the original template width and height, respectively. coef h coef These represent the scaling factors for the template width and height, respectively.

[0056] S27: Match the region of interest image described in step S1, set a confidence threshold, and record the coordinates of points whose scores are greater than the confidence threshold;

[0057] S28: If no traffic sign is matched, discard the current frame image data and jump to step S1 to reacquire the next frame road image data; if a traffic sign is matched, cluster the recorded coordinate points using an unsupervised clustering algorithm to obtain the upper left corner coordinates of the traffic sign location, and calculate the traffic sign positioning box by combining the template scaling scale.

[0058] S29: Based on the traffic sign positioning frame coordinates calculated above, crop the corresponding area of ​​the original image to obtain the traffic sign image.

[0059] Furthermore, in step S28, if a traffic sign is matched, the specific steps for locating the traffic sign are as follows:

[0060] S281: Let the set of coordinate sample points to be clustered be D = {X1, X2, ..., X...} n The clustering steps are as follows:

[0061] (1) Set parameters (ε, MinPts) to describe the density of the neighborhood sample distribution;

[0062] (2) Initialize the core object collection Initialize the number of clusters k = 0;

[0063] (3) Traverse the sample points X in set D i If |N ε (X i If X > MinPts, then X will be... i Added to the core object set Ω, where |N ε (X i | represents the condition that satisfies set N ε (X i The number of subsets of ), which is defined as shown in equation (16):

[0064] N ε (X i )={X j ∈D|distance(X i ,X j )≤ε} (16)

[0065] Where, distance(X) i ,X j ) represents sample X i X j The distance between them;

[0066] (4) If all elements in the core object set Ω have been visited, the algorithm ends; otherwise, proceed to step (5).

[0067] (5) Initialize the current seed set Randomly access an element o in the core object set Ω, mark the state of element o as visited and the category as k, and store the unvisited elements in the ε neighborhood of element o into the seed set Γ;

[0068] (6) If the seed set Then the current cluster C k Once generated and k = k + 1, proceed to step (4); otherwise, select an element s from the seed set Γ, first mark the state of element s as visited and the category as k, then determine whether the number of subsample sets satisfying equation (16) is greater than MinPts. If so, add the unvisited seed points in the subsample set to the seed set Γ, and mark the state of the elements contained in the subsample set as visited in the core object set Ω, then proceed to step (6).

[0069] S282: Calculate the average value of each category sample point after clustering, i.e., the final coordinate calculation formula is shown in the following formulas (17) and (18):

[0070]

[0071]

[0072] Where, x k ,y k The x and y coordinates represent the k-th cluster in the clustering results, where m represents the number of sample points in the k-th cluster. This represents the x and y coordinates of the i-th sample point in a k-category cluster;

[0073] S283: Based on the clustering results, the coordinates of the upper left corner and the lower right corner of the traffic sign in the detected road image are obtained, thus obtaining the final positioning result of the sign. The calculation formulas for the lower right corner coordinates of the traffic sign are shown in (19) and (20):

[0074] x max =x min +w new (19)

[0075] y max =y min +h new (20)

[0076] Where, x max ,y max The coordinates of the lower right corner of the location traffic sign are x and x respectively. min ,y min Let w represent the coordinates of the top left corner of the traffic sign obtained from clustering. new ,h new The width and height of the scaled template are calculated by equations (14) and (15).

[0077] Furthermore, step S3 specifically includes the following steps:

[0078] S31: Split the training dataset and save it to the local disk;

[0079] S32: Building a convolutional neural network model, specifically including:

[0080] S321: Define the input layer;

[0081] Determine the size and number of channels of the input image;

[0082] S322: Add a convolutional layer;

[0083] Add convolutional layers to the network according to the designed network structure;

[0084] S323: Add activation function;

[0085] Add activation functions according to the network model structure;

[0086] S324: Add a max pooling layer;

[0087] At the end of all convolutional blocks, a max pooling layer is added to compress the feature map into a single feature vector.

[0088] S325: Add a fully connected layer;

[0089] After the max pooling layer, add a fully connected layer to output the classification result; the shape of the output layer is determined according to the specific task requirements.

[0090] S326: Add a softmax layer;

[0091] After the fully connected layer, a softmax layer is added to calculate the probability of each classification result. The calculation formula is shown in (21):

[0092]

[0093] Where, x i f(x) is the i-th input to the softmax layer. i ) represents the i-th output of the softmax layer, x j This is the j-th input to the softmax layer;

[0094] S327: Compiler model;

[0095] Combine the network structure, loss function, and optimization algorithm to compile the model; select the optimization algorithm and specify the learning rate and weight decay.

[0096] S328: Training model;

[0097] Using the dataset, the model parameters are optimized through the backpropagation algorithm. During training, the model parameters are iteratively optimized to allow the model to gradually learn and improve its performance, thereby obtaining the optimal weights.

[0098] Furthermore, the activation function is a nonlinear function to prevent model overfitting, and its calculation formula is shown in (22):

[0099]

[0100] Where x is the independent variable of the input function, and Act(x) is the activation function.

[0101] Furthermore, step S4 specifically includes the following steps:

[0102] S41: Reconstruct a 3×3 convolutional layer by padding the convolutional layer around the 1×1 convolutional layer branch with 0s;

[0103] S42: The 3×3 convolution operator and the BN operator are combined, and the conversion formula is shown in the following (23):

[0104]

[0105] Where BN(Conv(x)) represents the fusion result of the Conv2d operator and the BN operator, γ represents the scaling factor of the BN layer, β represents the translation factor of the BN layer, and σ2 ε represents the variance of the input BN layer samples, μ represents the mean of the input BN layer samples, ε is a very small constant that prevents the denominator from being 0, W(x) represents the Conv2d layer weights, and ε represents the bias of the Conv2d layer.

[0106] S43: Transform the BN branch (which only has BN) into a 3×3 convolution. The implementation process is as follows:

[0107] Construct a 3×3 convolutional layer that only performs identity mapping, and then fuse it with the BN layer according to step S42;

[0108] S44: Merge the multi-branch structure by stacking the 3×3 convolutional layers after each branch transformation and merging them into a single-branch structure.

[0109] A traffic sign recognition system for active safety of commercial vehicles as described above is characterized by comprising an on-board camera device, an embedded system, a display screen, a warning device, a bus system, a vehicle control unit, and a motor control unit installed on the detection vehicle;

[0110] The vehicle-mounted camera device is used to capture panoramic road image data. It is installed at the front of the detection vehicle to obtain road information around the vehicle, and is connected to the embedded system via a USB interface to transmit the acquired image data to the embedded system for processing.

[0111] The embedded system is used to process image data and run a traffic sign recognition method; the embedded system is connected to the display screen via an HDMI video output interface to display the recognition results and related information on the display screen; the embedded system is connected to the warning device via a GPIO port to convey warning information to the driver; the embedded system communicates with the vehicle control unit via the bus system.

[0112] The display screen is used to show the recognition results and related information to the driver inside the vehicle;

[0113] The warning device is used to issue a warning signal to the driver, which may be an audible prompt, a flashing light, or a vibration device, to remind the driver to pay attention to traffic signs on the road.

[0114] The bus system is used to realize data exchange and communication between various devices, and uses the CAN bus inside the car to issue control commands to the car control unit.

[0115] The vehicle control unit is used to receive bus transmission commands and send control signals to the motor control unit;

[0116] The motor control unit is used to receive commands from the vehicle control unit to actively decelerate or brake the vehicle.

[0117] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0118] This invention uses an onboard camera device installed at the front of the vehicle to capture panoramic road images. It then uses HSV spatial template matching to locate and crop traffic signs. A neural network is applied to recognize the cropped images, and multi-branch fusion is performed during the recognition stage. This reduces the computational load of model inference without changing the model's accuracy, thereby reducing the inference time on the embedded end. If a violation is detected, the vehicle will be automatically braked, reducing the probability of safety hazards caused by drivers ignoring traffic signs. Attached Figure Description

[0119] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0120] In the attached diagram:

[0121] Appendix Figure 1 This is a flowchart of the traffic sign recognition method of the present invention;

[0122] Appendix Figure 2 This is a structural block diagram of the traffic sign recognition system of the present invention;

[0123] Appendix Figure 3 This is a flowchart illustrating the creation process of the traffic sign template library for this invention.

[0124] Appendix Figure 4 This is a flowchart of the traffic sign location clustering process of the present invention;

[0125] Appendix Figure 5 This invention relates to a multi-branch module structure diagram of a convolutional neural network model;

[0126] Appendix Figure 6 This is a structural diagram of the traffic sign recognition network training model of the present invention;

[0127] Appendix Figure 7 This is a structural diagram of the traffic sign recognition network inference model of the present invention. Detailed Implementation

[0128] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0129] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0130] The present invention will now be described in detail with reference to the accompanying drawings and exemplary embodiments.

[0131] Example 1

[0132] This embodiment discloses a traffic sign recognition method for active safety of commercial vehicles, used to identify traffic signs on the road during the movement of commercial vehicles, thereby ensuring the driving safety of commercial vehicles. Figure 1 As shown, the traffic identification method includes the following steps:

[0133] S1. Capture panoramic road image information through the vehicle-mounted camera device, preprocess the panoramic road image information, and extract the region of interest image by setting a threshold in the color transformation space.

[0134] S2. Match the above-mentioned region of interest image with the traffic sign template library image, and locate the sign image in the panoramic image of the road sign based on the matching result;

[0135] S3. Design and build a neural network model, train the neural network model on the sampled traffic sign dataset, and obtain the optimal traffic sign recognition model;

[0136] S4. Merge the multi-branch structure in the above optimal traffic sign recognition model to obtain the reshaped model;

[0137] S5. Classify and identify the sign images described in step S2. After normalizing the sign images, input them into the reshaping model in step S4. Determine the type of traffic sign based on the output results.

[0138] In one embodiment, step S1, obtaining the region of interest image specifically includes the following steps:

[0139] S11: Capture panoramic road image information through vehicle-mounted camera device, capturing a panoramic road image with a height of 1280 and a width of 720;

[0140] S12: Image cropping retains the top image at 0.6 times its height, meaning the cropped image has a height and width of 768 and 720 respectively, removing invalid information such as the car head at the bottom of the image;

[0141] S13: Perform smoothing filtering on the upper image described in step S12. In this embodiment, the weighting function of the filtering algorithm is a constant 1, and the filtering formula is as follows:

[0142]

[0143] Where g(x,y) is the original image, and a and b are integers and

[0144] S14: The cropped image is transformed using the HSV color space to obtain the final preprocessed image. The color threshold ranges set in this embodiment are as follows:

[0145] The H threshold is set to: (0,25) and (156,180), yellow (26,34), and blue (100,124);

[0146] The S threshold is set to (43, 255);

[0147] The V threshold is set to (46, 255).

[0148] In one embodiment, the process of creating the traffic sign template library in step S21 is as follows: Figure 3 As shown, the specific process is as follows:

[0149] S211: This embodiment selects 36 types of traffic signs, and chooses clear and recognizable sign images for different sign types;

[0150] S212: Convert RGB format to HSV format;

[0151] S213: Extract the red, yellow, or blue regions in the HSV color space according to the threshold set in step 1 to obtain the target color region;

[0152] S214: Obtain a template image from the extracted target color region, the template image representing the shape and color features of a specific type of logo;

[0153] S215: Combine and compile the processed and extracted template images to form a template library containing various types of logos.

[0154] S22: Set the scaling factor list for the template image width and height. In this embodiment, the scaling factor list is selected as [0.5, 3]. The interval increases by 0.1, and 26 scaling factors are selected. The template height and width are scaled by the same multiple.

[0155] S23: Select the coefficient values ​​in the list one by one to perform template matching between the aforementioned template image and the aforementioned HSV segmentation image;

[0156] S24: During the template matching process, the matching score corresponding to each set of coefficients is recorded. The score is calculated using the following formula (13):

[0157]

[0158] Where Score represents the matching score between the two, T' represents the template pixel value, and I' represents the target image pixel value. The value is 1 when the template and the sliding window match perfectly, and -1 when they do not match at all.

[0159] S25: Select the set of coefficients with the highest scores from all matching scores as the final scaling factor for the template.

[0160] S26: Scale the height and width of the template according to the scaling factor;

[0161] S27: Match the region of interest image extracted in step S1. In this embodiment, the confidence level is set to 0.75, and the coordinates of the points with scores greater than the confidence threshold are recorded.

[0162] S28: If no traffic sign is matched, discard the current frame image data and proceed to step S1 to acquire the next frame road image data; if a traffic sign is matched, refer to... Figure 4 The clustering process involves clustering the coordinate points of the records using a clustering algorithm to obtain the coordinates of the upper left corner of the traffic sign's location, and then calculating the traffic sign's location frame by combining the template scaling scale.

[0163] S29: Based on the traffic sign positioning frame coordinates calculated above, crop the corresponding area of ​​the original image to obtain the traffic sign image.

[0164] In one embodiment, the construction of the neural network model in step S3, Figure 5 This is a diagram of the multi-branch module structure of the model in this embodiment. Figure 6 The diagram shows the structure of the neural network model during the training phase. The model is trained on a sampled traffic sign dataset to obtain the optimal traffic sign recognition model.

[0165] Specifically, the following steps are included:

[0166] S31: Divide the training dataset and save it to the local disk, naming each category folder after the category name;

[0167] S32: Building a convolutional neural network model, specifically including:

[0168] S321: Define the input layer

[0169] Determine the size and number of channels of the input image. The input image is normalized to a size of 224*224 and is in RGB three-channel format.

[0170] S322: Add a convolutional layer

[0171] Reference Figure 6The network structure is as follows, with convolutional layers added. Convolutional layer 601 has parameters set to a kernel size of 3*3, a stride of 2, and padding of 1, followed by a batch normalization (BN) layer; convolutional layer 602 has parameters set to a kernel size of 3*3, a stride of 2, and padding of 0, followed by a BN layer; the structure of multi-branch modules 1 to 21 is as follows. Figure 5 As shown, in module 501, the convolutional kernel size is 1*1, the stride is 2, the padding is 0, and it is followed by a BN layer; in module 502, the convolutional kernel size is 3*3, the stride is 2, the padding is 0, and it is followed by a BN layer.

[0172] S323: Add activation function

[0173] Reference Figure 5 In a multi-branch module structure, after adding ConvBN module 501, ConvBN module 502, and BN module 503, an activation function is input. The expression of this activation function is:

[0174]

[0175] S324: Add a max pooling layer

[0176] After the multi-branch block 21, a max pooling layer is added to compress the feature map into a 320*768 feature vector.

[0177] S325: Add a fully connected layer

[0178] After the max pooling layer, a fully connected layer is added to output the classification results. In this embodiment, a total of 36 traffic signs are selected, and the recognition results are one of the 36 traffic signs or other categories, that is, a total of 37 categories. The output shape of the fully connected layer is 1*37.

[0179] S326: Add a softmax layer

[0180] The softmax layer is used to calculate the probability that the recognition result is one of 37 categories, with a shape of 1*37 and an arithmetic sum of 1.

[0181] S327: Compiler Model

[0182] The network structure, loss function, and optimization algorithm are combined to compile the model. An optimization algorithm is selected, and hyperparameters such as learning rate and weight decay are specified. In this embodiment, the optimization algorithm is stochastic gradient descent (SGD), the learning rate is set to 0.01, the weight decay is set to a certain value, and the loss function is cross-entropy loss, calculated using the following formula:

[0183]

[0184] Where L is the loss value, N is the number of samples, M is the number of classes, and y ic p is a sign function; it takes the value 1 if the true class of sample i is c, and 0 otherwise. ic Let be the predicted probability that observed sample i belongs to category c.

[0185] S328: Training Model

[0186] Using the dataset, the model parameters are optimized through backpropagation. During training, the model parameters are iteratively optimized, allowing the model to gradually learn and improve its performance, ultimately obtaining the optimal weights.

[0187] In one embodiment, in step S4, the model structure is reshaped, and the reshaped structure is as follows: Figure 7 As shown, the specific steps are as follows:

[0188] S41: Fill the 1×1 convolutional layer in the multi-branch module with zeros and reconstruct it into a 3×3 convolutional layer;

[0189] S42: Merge the 3×3 convolution operator and BN operator in the multi-branch module, using the following conversion formula:

[0190]

[0191] Where BN(Conv(x)) represents the fusion result of the Conv2d operator and the BN operator, γ represents the scaling factor of the BN layer, β represents the translation factor of the BN layer, and σ 2 represents the variance of the input BN layer samples, represents the mean of the input BN layer samples, is a very small constant to prevent the denominator from being 0, W(x) represents the weight of the Conv2d layer, and represents the bias of the Conv2d layer.

[0192] S43: In the multi-branch module, construct a 3×3 convolutional layer that only performs identity mapping on the branch with BN, and fuse it with the BN layer according to step S42;

[0193] S44: The 3×3 convolutional layers fused and transformed from S41, S42, and S43 are added together and merged into a single-branch structure, such as... Figure 7 As shown.

[0194] Example 2

[0195] Figure 2 This is a schematic diagram of a traffic sign recognition system for active safety of commercial vehicles provided in Embodiment 2 of the present invention. The system includes: a camera device 201, an embedded system 202, a display screen 203, a warning device 204, a bus system 205, a vehicle control unit 206, and a motor braking unit 207.

[0196] The camera device 201 is used to capture panoramic road image data. In this embodiment, the camera is installed on the dashboard in front of the passenger side of the vehicle and is connected to the embedded system via a USB interface. The acquired image data is transmitted to the embedded system 202 for processing. In this embodiment, a camera with a resolution of 1280*720 is used, and the height and width of the acquired image are 1280 and 720, respectively.

[0197] The embedded system 202 is used to run the road traffic sign recognition algorithm and process the image data captured by the camera. In this embodiment, the embedded system includes components such as a processor, memory and storage devices. It is connected to the display screen 203 through an HDMI video output interface to display the recognition results and related information on the display screen 203. It is connected to the warning device 204 through a GPIO port to convey warning information to the driver. It communicates with the vehicle control unit 206 in the vehicle through the bus system 205.

[0198] Display screen 203 is used to display the recognition results and related information to the driver inside the vehicle, such as the type of traffic sign recognized and warning information.

[0199] Warning device 204 is used to convey important traffic sign identification information to the driver through sound or flashing lights, so as to remind the driver to pay attention to road rules and safety.

[0200] The bus system 205 is used to transmit data and commands. In this embodiment, the CAN bus inside the car is used to issue control commands to the car control unit 206.

[0201] The vehicle control unit 206 receives bus transmission commands and sends control signals to the motor control unit 207.

[0202] The motor control unit 207 receives commands from the vehicle control unit to actively decelerate or brake the vehicle.

[0203] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A traffic sign recognition method for active safety of commercial vehicles, characterized in that, The specific steps of the identification method are as follows: S1. Capture panoramic road image information through the vehicle-mounted camera device, preprocess the panoramic road image information, and extract the region of interest image by setting a threshold in the color transformation space. S2. Match the above-mentioned region of interest image with the traffic sign template library image, and locate the sign image in the road panoramic image based on the matching result; S3. Design and build a neural network model. The neural network model contains several multi-branch blocks, and each multi-branch block contains 1 parallel... 1 convolutional layer, 3 The neural network model is trained on a sampled traffic sign dataset with three convolutional layers and BN layers, and several branch blocks are connected in series to obtain the optimal traffic sign recognition model. S4. Merge several multi-branch blocks from the above optimal traffic sign recognition model. 1 convolutional layer, 3 Three convolutional layers and BN layers are used to obtain a new reshaped model; S5. Classify and identify the sign images described in step S2. After normalizing the sign images, input them into the reshaping model in step S4. Determine the traffic sign type based on the output results. Step S4 specifically includes the following steps: S41: Reconstruct a 3×3 convolutional layer by padding the convolutional layer around the 1×1 convolutional layer branch with 0s; S42: The 3×3 convolution operator and the BN operator are combined, and the conversion formula is shown in (23) below: (23) in, This represents the fusion result of the Conv2d operator and the BN operator. Indicates the scaling factor of the BN layer. Indicates the translation coefficient of the BN layer. ε represents the variance of the input BN layer samples, μ represents the mean of the input BN layer samples, and ε is a very small constant used to prevent the denominator from being zero. represents the Conv2d layer weights, and b represents the Conv2d layer bias. S43: Transform the BN branch (which only has BN) into a 3×3 convolution. The implementation process is as follows: Construct a 3×3 convolutional layer that only performs identity mapping, and then fuse it with the BN layer according to step S42; S44: Merge the multi-branch structure by stacking the 3×3 convolutional layers after each branch transformation and merging them into a single-branch structure.

2. The traffic sign recognition method for active safety of commercial vehicles as described in claim 1, characterized in that, In step S1, obtaining the region of interest image specifically includes the following steps: S11: The vehicle-mounted camera device acquires a road image with a width of width pixels and a height of height pixels; S12: Retain the upper image of the captured road image at a height of h times, and the width and height of the cropped image are respectively... It can be calculated using the following formulas (1) and (2): (1) (2) S13: Perform smoothing filtering on the upper image described in step S12. Assuming a window of size m×n is selected to filter the upper image, the calculation formula at the center pixel (x,y) of the filtering window is as follows (3): (3) in, It is a weighting function, where r and s are the horizontal and vertical coordinate offsets of the current pixel in the filtering window relative to the center pixel of the filtering window, respectively. Different weights are assigned to the weighting function according to the actual engineering needs. It is the original image, where a and b are integers and ; S14: Perform HSV spatial transformation on the image after filtering in step S13 above. Assuming that the red channel value is R, the green channel value is G, and the blue channel value is B at a certain point in the current image, the H, S, and V values ​​after HSV spatial transformation can be obtained through the following specific transformation steps: S141: Extract the red channel value R, green channel value G, and blue channel value B from the RGB image; S142: Normalize R, G, and B to obtain The calculation formulas are shown in equations (4), (5) and (6) below: (4) (5) (6) S143: Calculate the maximum channel value and minimum channel value The calculation formulas are shown in equations (7) and (8) below; (7) (8) S144: The hue (H), saturation (S), and brightness (V) of an RBG format image converted to HSV space are calculated using the following formulas (9), (10), and (11): (9) (10) (11) S15, set the thresholds of H, S, and V in step S14 to obtain the mask, multiply the mask with the upper image in step S13 to obtain the region of interest image, and calculate the formula as shown in equation (12): (12) in, This indicates obtaining the pixel value of the image of interest at coordinates (x, y). This represents the pixel value at coordinates (x, y) in the original image. This represents the pixel value of the mask at coordinates (x, y).

3. The traffic sign recognition method for active safety of commercial vehicles as described in claim 2, characterized in that, Step S2 specifically includes the following steps: S21: Create a traffic sign template library, which includes the following steps: S211: For each type of mark to be detected, select a clear and identifiable mark image as a sample; S212: Convert the selected logo image from RGB format to HSV format. The conversion process is completed according to the above steps S142, S143 and S144 so that subsequent processing can be performed in the HSV color space. S213: In the HSV color space, based on a set threshold, an extraction operation is performed on the red, yellow, or blue regions to obtain the target color region; S214: Obtain a template image from the extracted target color region, the template image representing the shape and color features of a specific type of logo; S215: Combine the processed and extracted template images to form a template library containing various types of logos; S22: Set a list of width and height scaling factors for scaling the template image. This list contains multiple sets of width and height scaling factors. S23: Perform template matching on the aforementioned template library images and the aforementioned HSV segmentation images; for each set of width and height scaling factors, during the matching process, scale the template images according to the current scaling factors, and then match them with the HSV segmentation images; S24: During the template matching process, the matching score corresponding to each set of coefficients is recorded. The score is calculated using the following formula (13): (13) Wherein, Score represents the matching score between the two. Represents the template pixel value. This represents the pixel value of the target image. The value is 1 when the template and the sliding window match perfectly, and -1 when they do not match at all. S25: Select the set of coefficients with the highest scores from all matching scores as the final scaling factor for the template; S26: Scale the template height and width according to the aforementioned scaling factors, and calculate the scaled width and height of the template using formulas (14) and (15): (14) (15) in, , The width and height of the template after scaling are shown separately. , These represent the original template width and height, respectively. , These represent the scaling factors for the template width and height, respectively. S27: Match the region of interest image described in step S1, set a confidence threshold, and record the coordinates of points whose scores are greater than the confidence threshold; S28: If no traffic sign is matched, discard the current frame image data and jump to step S1 to reacquire the next frame road image data; if a traffic sign is matched, cluster the recorded coordinate points using an unsupervised clustering algorithm to obtain the upper left corner coordinates of the traffic sign location, and calculate the traffic sign positioning box by combining the template scaling scale. S29: Based on the traffic sign positioning frame coordinates calculated above, crop the corresponding area of ​​the original image to obtain the traffic sign image.

4. The traffic sign recognition method for active safety of commercial vehicles as described in claim 3, characterized in that, In step S28, if a traffic sign is matched, the specific steps for locating the traffic sign are as follows: S281: Let the set of coordinate sample points to be clustered be... The clustering steps are as follows: (1) Set parameters (ε, MinPts) to describe the density of the neighborhood sample distribution; (2) Initialize the core object collection Ω= Initialize the number of clusters k=0; (3) Traverse the sample points X in set D i If satisfied Then X i Added to the core object set Ω, where Represents a set that satisfies The number of subsets, defined as shown in equation (16): (16) in, Indicates sample X i X j The distance between them; (4) If all elements in the core object set Ω have been visited, the algorithm ends; otherwise, proceed to step (5). (5) Initialize the current seed set Γ= Randomly access an element o in the core object set Ω, mark the state of element o as visited and the category as k, and store the unvisited elements in the ε neighborhood of element o into the seed set Γ; (6) If the seed set Γ= Then the current cluster C k Once generated and k = k + 1, proceed to step (4); otherwise, select an element s from the seed set Γ, first mark the state of element s as visited and the category as k, then determine whether the number of subsample sets satisfying equation (16) is greater than MinPts. If so, add the unvisited seed points in the subsample set to the seed set Γ, and mark the state of the elements contained in the subsample set as visited in the core object set Ω, then proceed to step (6). S282: Calculate the average value of each category sample point after clustering, i.e., the final coordinate calculation formula is shown in the following formulas (17) and (18): (17) (18) in, The x and y coordinates represent the k-th cluster in the clustering results, where m represents the number of sample points in the k-th cluster. This represents the x and y coordinates of the i-th sample point in a k-category cluster; S283: Based on the clustering results, the coordinates of the upper left corner and the lower right corner of the traffic sign in the detected road image are obtained, thus obtaining the final positioning result of the sign. The formulas for calculating the lower right corner coordinates of the traffic sign are shown in (19) and (20): (19) (20) in, These represent the coordinates of the lower right corner of the location traffic sign. These represent the coordinates of the top-left corner of the traffic sign obtained from clustering. The width and height of the scaled template are calculated by equations (14) and (15).

5. A traffic sign recognition method for active safety of commercial vehicles as described in claim 1, characterized in that, Step S3 specifically includes the following steps: S31: Split the training dataset and save it to the local disk; S32: Building a convolutional neural network model, specifically including: S321: Define the input layer; Determine the size and number of channels of the input image; S322: Add a convolutional layer; Add convolutional layers to the network according to the designed network structure; S323: Add activation function; Add activation functions according to the network model structure; S324: Add a max pooling layer; At the end of all convolutional blocks, a max pooling layer is added to compress the feature map into a single feature vector. S325: Add a fully connected layer; After the max pooling layer, add a fully connected layer to output the classification result; the shape of the output layer is determined according to the specific task requirements. S326: Add a softmax layer; After the fully connected layer, a softmax layer is added to calculate the probability of each classification result. The calculation formula is shown in (21): (21) in, For the i-th input of the softmax layer, This is the i-th output of the softmax layer. This is the j-th input to the softmax layer; S327: Compiler model; Combine the network structure, loss function, and optimization algorithm to compile the model; select the optimization algorithm and specify the learning rate and weight decay. S328: Training model; Using the dataset, the model parameters are optimized through the backpropagation algorithm. During training, the model parameters are iteratively optimized to allow the model to gradually learn and improve its performance, thereby obtaining the optimal weights.

6. A traffic sign recognition method for active safety of commercial vehicles as described in claim 5, characterized in that, The activation function is a nonlinear function to prevent the model from overfitting, and its calculation formula is shown in (22): (22) Where x is the independent variable of the input function, This is the activation function.

7. A recognition system for a traffic sign recognition method for active safety of commercial vehicles as described in any one of claims 1 to 6, characterized in that, This includes onboard camera devices, embedded systems, displays, warning devices, bus systems, vehicle control units, and motor control units installed on the inspection vehicle; The vehicle-mounted camera device is used to capture panoramic road image data. It is installed at the front of the detection vehicle to obtain road information around the vehicle, and is connected to the embedded system via a USB interface to transmit the acquired image data to the embedded system for processing. The embedded system is used to process image data and run the traffic sign recognition method as described in claims 1 to 6; the embedded system is connected to the display screen via an HDMI video output interface to display the recognition results and related information on the display screen; the embedded system is connected to the warning device via a GPIO port to convey warning information to the driver; The embedded system communicates with the vehicle control unit via the bus system; The display screen is used to show the recognition results and related information to the driver inside the vehicle; The warning device is used to issue a warning signal to the driver, which may be an audible prompt, a flashing light, or a vibration device, to remind the driver to pay attention to traffic signs on the road. The bus system is used to realize data exchange and communication between various devices, and uses the CAN bus inside the car to issue control commands to the car control unit. The vehicle control unit is used to receive bus transmission commands and send control signals to the motor control unit; The motor control unit is used to receive commands from the vehicle control unit to actively decelerate or brake the vehicle.

Images