Traffic sign detection method and device, electronic equipment and storage medium

Abstract

The invention provides a traffic sign detection method and device, electronic equipment and a storage medium, and is applied to the technical field of vehicles, and the method comprises the steps: obtaining a vehicle coordinate and a target image feature of a vehicle driving region image, and inputting the target image feature into a traffic sign detection model, obtaining sign information and sign coordinates of traffic signs in the vehicle driving area image, inputting target image features into a traffic sign semantic segmentation model to obtain a foreground segmentation result of the vehicle driving area image, and inputting the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain target sign coordinates of the traffic signs; obtaining the distance between the traffic sign and the vehicle according to the target sign coordinate and the vehicle coordinate, and outputting the sign information of the traffic sign under the condition that the distance between the traffic sign and the vehicle meets the distance judgment condition, so that the traffic sign related to the vehicle can be accurately obtained.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a traffic sign detection method, device, electronic equipment and storage medium. Background Art

[0002] In the field of intelligent driving, vehicles can detect traffic signs on the road while driving, control vehicle operation based on the detected traffic signs, or present the detected traffic signs to the driver through audio, video, pictures, etc.

[0003] In the related art, after a traffic sign is detected, the detected traffic sign is directly outputted, and these traffic signs may include traffic signs irrelevant to the vehicle, which may bring inconvenience to the intelligent control of the vehicle and the operation of the driver.

[0004] That is, in the related art, there is a lack of a method for accurately and in real time screening traffic signs related to vehicles. Summary of the invention

[0005] In view of the above problems, embodiments of the present invention provide a traffic sign detection method, device, electronic device and storage medium that overcome the above problems or at least partially solve the above problems.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, an embodiment of the present application discloses a traffic sign detection method, comprising:

[0008] Obtaining vehicle coordinates and target image features of the vehicle driving area image;

[0009] Inputting the target image features into a traffic sign detection model to obtain the sign information and sign coordinates of the traffic signs in the vehicle driving area image, and inputting the target image features into a traffic sign semantic segmentation model to obtain a foreground segmentation result of the vehicle driving area image;

[0010] Inputting the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign;

[0011] The distance between the traffic sign and the vehicle is obtained according to the target sign coordinates and the vehicle coordinates, and the sign information of the traffic sign is output when the distance between the traffic sign and the vehicle meets a distance judgment condition.

[0012] In a second aspect, an embodiment of the present application discloses a traffic sign detection device, comprising:

[0013] A first acquisition module is used to acquire vehicle coordinates and target image features of an image of a vehicle driving area;

[0014] The second acquisition module is used to input the target image features into a traffic sign detection model to obtain the sign information and sign coordinates of the traffic signs in the vehicle driving area image, and input the target image features into a traffic sign semantic segmentation model to obtain the foreground segmentation result of the vehicle driving area image;

[0015] A third acquisition module is used to input the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign;

[0016] The fourth acquisition module is used to obtain the distance between the traffic sign and the vehicle according to the target sign coordinates and the vehicle coordinates, and output the sign information of the traffic sign when the distance between the traffic sign and the vehicle meets the distance judgment condition.

[0017] In a third aspect, an embodiment of the present application discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0018] In a fourth aspect, an embodiment of the present application discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present application discloses a vehicle, comprising the traffic sign detection device as described in the second aspect, or the electronic device as described in the third aspect.

[0020] In this embodiment, after obtaining the vehicle target image features, the vehicle target image features are input into the traffic sign detection model and the traffic sign semantic segmentation model, that is, the traffic sign detection model and the traffic sign semantic segmentation model share the target image features, instead of obtaining the image features as model input parameters for the two models respectively, thereby saving computing resources, improving processing efficiency, and improving the timeliness and accuracy of traffic sign detection. Based on the multi-layer perceptron, the detection results of the traffic sign detection model and the traffic sign semantic segmentation model are fused and processed to obtain accurate detection effects. The distance between the traffic sign and the vehicle is obtained according to the target sign coordinates and the vehicle coordinates. When the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign information of the traffic sign is output, which is equivalent to adding restriction conditions to the detection results of the traffic sign inspection model, and judging whether the current scene needs to consider the traffic sign according to the restriction conditions, so as to perform subsequent operations according to the selected traffic signs. This implementation can filter out traffic signs that are not related to the vehicle, filter out traffic signs related to the vehicle, improve the effectiveness of traffic sign detection, and improve the processing efficiency of subsequent processing based on the output traffic signs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the steps of a traffic sign detection method provided by an embodiment of the present invention;

[0022] Figure 2 is a flowchart of the steps of another landmark coordinate detection model provided by an embodiment of the present invention;

[0023] Figure 3 is a flow chart of steps of a traffic sign detection model provided by an embodiment of the present invention;

[0024] Figure 4 is a flow chart of the steps of a traffic sign detection method provided by an embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of a traffic sign screening method provided by an embodiment of the present invention;

[0026] Figure 6 is a flowchart of another traffic sign detection method provided by an embodiment of the present invention;

[0027] Figure 7 is a block diagram of a traffic sign detection device provided by an embodiment of the present invention;

[0028] Figure 8 is a block diagram of an electronic device provided by an embodiment of the present invention;

[0029] Fig. 9 This is a block diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0031] Figure 1 The traffic sign detection method provided by the embodiment of the present application is shown, referring to Figure 1 , the method may include the following steps:

[0032] Step 101, obtaining vehicle coordinates and target image features of the vehicle driving area image.

[0033] For example, the vehicle coordinates may be obtained through a positioning system, for example, the vehicle coordinates may be obtained through a global positioning system (GPS).

[0034] For example, an image acquisition device provided in the vehicle can be used to obtain an image of the vehicle driving area. For example, the image of the vehicle driving area can be obtained through a camera. The image of the vehicle driving area can be input into an image feature extraction model to obtain target image features of the vehicle driving area.

[0035] For example, the image feature extraction network can be composed of a deep residual network (Residual Network-101, ResNet-101) and a feature pyramid network (Feature Pyramid Networks, FPN). Combining ResNet-101 and feature pyramid network FPN for feature extraction can perform multi-scale feature fusion on the image features of the vehicle driving area to obtain more detailed information, thereby improving the model's detection accuracy for small traffic signs or long-distance traffic signs.

[0036] Step 102, input the target image features into the traffic sign detection model to obtain the sign information and sign coordinates of the traffic signs in the vehicle driving area image, and input the target image features into the traffic sign semantic segmentation model to obtain the foreground segmentation result of the vehicle driving area image.

[0037] For example, refer to Figure 2, a red, green, blue (RGB) image of the vehicle driving area can be obtained, and the image can be input into a feature extraction network for extracting image features to obtain target image features, and then the obtained target image features can be input into a traffic sign detection model to obtain the sign information of the traffic sign in the vehicle driving area image (for example, the traffic type shown in the figure) and the bounding box position correction result. The bounding box position correction result includes (x, y, w, h, θ), where x, y, w, h, θ respectively represent the X-axis coordinate, Y-axis coordinate, length, width and angle of the target box relative to the preset horizontal line of the target box used to annotate the traffic sign.

[0038] Furthermore, Figure 3 FIG. 4 is a schematic diagram showing a traffic sign detection model provided in this embodiment. Figure 3 , the RGB image of the driving area is input into the feature extraction network to obtain the target image features, the target image features are input into the Region Proposal Network (RPN) to obtain the proposal box, and then the target image features and the proposal box are input into the Region of Interest Alignment (RoI Align) network to obtain the sign category and bounding box regression results of the traffic sign. Among them, in the traffic sign detection model, RoIAlign is used instead of RoIPooling for the pooling layer in the model to reduce the quantization error in the candidate box size transformation operation and improve the accuracy of the detection results of the traffic sign detection model.

[0039] Step 103, inputting the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign;

[0040] Reference Figure 2 ,The bounding box position correction result and the pixel-level foreground segmentation result are input into the multi-layer perceptron to obtain the target sign coordinates of the traffic sign.

[0041] Reference Figure 2 Based on the traffic sign detection model, a traffic sign semantic segmentation model is added as a parallel branch of the traffic sign detection model. The traffic sign semantic segmentation model is used to predict the image segmentation of traffic signs. The traffic sign semantic segmentation model and the traffic sign detection model share a feature extraction network, which can save computing resources.

[0042] For example, a lightweight Mini-DeepLab module can be used as the main part of the traffic sign semantic segmentation model. According to the traffic semantic segmentation model with the lightweight Mini-DeepLab module as the theme, the target image features are processed to ensure the timeliness and accuracy of the model processing.

[0043] The prediction results of the two branches are fused and refined through an information fusion and refinement module based on a multilayer perceptron (MLP), thereby obtaining a detection result with higher precision and accuracy.

[0044] Furthermore, the sign coordinates and foreground segmentation results are input into the multi-layer perceptron and then into the MLP to obtain the fused and refined bounding box position detection results (x1, y1, w1, h1, θ1). Among them, x1, y1, w1, h1, θ1 respectively represent the fused and refined target box for marking traffic signs, the X-axis coordinates and Y-axis coordinates in the vehicle coordinate system, and the length, width, and angle of the target box relative to the preset horizontal line.

[0045] For example, Zhang's calibration method can be used to convert the bounding box position detection result (x1, y1, w1, h1, θ1) from the pixel coordinate system to the camera coordinate system, and then from the camera coordinate system to the vehicle coordinate system. Zhang's calibration method is a camera calibration method. Through this camera calibration method, the target traffic representation coordinates of the traffic sign in the vehicle coordinate system are obtained.

[0046] Step 104, obtaining the distance between the traffic sign and the vehicle according to the target sign coordinates and the vehicle coordinates, and outputting the sign information of the traffic sign when the distance between the traffic sign and the vehicle meets the distance judgment condition.

[0047] For example, the distance judgment condition may be a condition set according to empirical data, and the distance judgment condition may include a lateral distance judgment condition and a longitudinal distance judgment condition. For example, according to the target sign coordinates and the vehicle coordinates, the lateral distance and the longitudinal distance between the traffic sign and the vehicle are obtained. When the lateral distance falls within a preset lateral distance range and the longitudinal distance falls within a preset longitudinal distance range, it is determined that the distance between the traffic sign and the vehicle meets the distance judgment condition, and the sign information of the traffic sign is output.

[0048] For example, the sign information includes the sign content of the traffic sign, and the sign content may include signs such as "watch out for pedestrians", "construction ahead", and "watch out for children".

[0049] With the development of artificial intelligence, computer vision technology based on deep learning has promoted the development of intelligent driving. In the field of visual perception, based on deep learning methods, during the driving process of the vehicle, robot vision can be used to replace human vision for target detection, so that the intelligent driving system can automatically analyze the scene and perceive the object. In the related art, the detection of traffic signs by intelligent driving visual perception technology has the following problems: intelligent driving vehicles usually need to be equipped with multi-task systems. Parallel processing of multiple tasks during driving will cause great pressure on computing resources, resulting in poor timeliness and accuracy in the detection process, which in turn causes the detection results to be unable to provide reliable support for subsequent tasks such as obstacle avoidance and path planning. In addition, the related art lacks a method that can accurately and real-timely screen out valid traffic signs related to the vehicle from multiple traffic signs detected in a variety of driving road scenes. The method of the related art makes it difficult for the intelligent driving system to respond accurately to the content shown by the traffic signs in a timely manner according to the valid traffic signs.

[0050] In this embodiment, after obtaining the vehicle target image features, the vehicle target image features are input into the traffic sign detection model and the traffic sign semantic segmentation model, that is, the traffic sign detection model and the traffic sign semantic segmentation model share the target image features, instead of obtaining the image features as model input parameters for the two models respectively, thereby saving computing resources, improving processing efficiency, and improving the timeliness and accuracy of traffic sign detection. Based on the multi-layer perceptron, the detection results of the traffic sign detection model and the traffic sign semantic segmentation model are fused and processed to obtain accurate detection effects. The distance between the traffic sign and the vehicle is obtained according to the target sign coordinates and the vehicle coordinates. When the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign information of the traffic sign is output, which is equivalent to adding restriction conditions to the detection results of the traffic sign inspection model, and judging whether the current scene needs to consider the traffic sign according to the restriction conditions, so as to perform subsequent operations according to the selected traffic signs. This implementation can filter out traffic signs that are not related to the vehicle, filter out traffic signs related to the vehicle, improve the effectiveness of traffic sign detection, and improve the processing efficiency of subsequent processing based on the output traffic signs. Among them, subsequent processing based on the output sign information may include, in an autonomous driving vehicle, controlling the operation of the vehicle according to the output sign information, and displaying the sign information to the driver through audio, pictures, etc. for the driver to use as a driving reference.

[0051] Figure 4 The traffic sign detection method provided by the embodiment of the present application is shown, referring to Figure 4 , the method may include the following steps:

[0052] Step 201, obtaining vehicle coordinates and vehicle driving area images.

[0053] The method of this step has been described in the aforementioned step 101 and will not be repeated here.

[0054] Step 202: Input the vehicle driving area image into a deep residual network to obtain a first image feature.

[0055] For example, the deep residual network may be ResNet-101.

[0056] Step 203: Input the first image feature into the feature pyramid network to obtain the target image feature.

[0057] For example, the feature pyramid network is an FPN network. The method of this step has been described in the aforementioned step 102 and will not be repeated here.

[0058] Step 204: Input the target image features into the region proposal network to obtain a suggestion box prediction result.

[0059] For example, according to the requirements of traffic sign detection performance in intelligent driving scenarios, this embodiment uses a faster region-based convolutional neural network (Faster Region-based Convolutional Neural Network, FasterR-CNN) as the model basis to construct a traffic sign detection model based on Faster R-CNN. The structure of the traffic sign detection model is as follows: Figure 3 shown.

[0060] Reference Figure 3 In the region proposal network of the traffic sign detection model, an anchor box mechanism with direction is set. The angles of the anchor box are set to ±15°, ±45° and ±75°. Based on the anchor box mechanism, accurate location information can be predicted.

[0061] Step 205 , input the target image features and the prediction results of the suggestion boxes into the region of interest alignment network to obtain the sign information and sign coordinates of the traffic sign in the image.

[0062] Among them, refer to Figure 3 , the target image features and the proposed box prediction results are input into the region of interest alignment network RoIAlign to obtain the sign information and sign coordinates of the traffic sign in the image. Among them, the sign coordinates are expressed in the form of a bounding box.

[0063] The method of this step has been described in the aforementioned step 102 and will not be repeated here.

[0064] Step 206: Input the target image features into the traffic sign semantic segmentation model to obtain the foreground segmentation result of the vehicle driving area image.

[0065] The method of this step has been described in the aforementioned step 102 and will not be repeated here.

[0066] Step 207, input the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign.

[0067] This embodiment can accurately obtain the target sign coordinates of traffic signs through the detection model that integrates the traffic sign semantic segmentation model and the traffic sign detection model. The traffic sign semantic segmentation model and the traffic sign detection model use the same feature extraction network to extract target image features, which improves the real-time performance of model prediction.

[0068] For example, the traffic sign semantic segmentation model can use the lightweight Mini-DeepLab module as the main part. Traffic signs can be extracted based on the Mini-DeepLab module, which can save computing resources and ensure real-time performance.

[0069] In the Faster R-CNN algorithm of the related technology, the pooling network is a RoIPooling pooling network. In this embodiment, the RoI Align pooling network is used to replace the RoIPooling pooling network, thereby improving the Faster R-CNN algorithm in the related technology. The improved Faster R-CNN algorithm is used as the backbone network of the traffic sign detection model, thereby improving the accuracy of the detection results of the traffic sign detection model.

[0070] In practical applications, driving usually requires a multi-task system. During driving, multiple perception tasks run in parallel, which will consume a lot of computing resources, resulting in a lag in traffic sign detection and low accuracy. In this embodiment, based on the method of steps 201 to 207, the target image features are extracted by inputting an image feature extraction model composed of a deep residual network and a feature pyramid network, and then the target image features are extracted and processed by the same image feature extraction model in combination with the traffic sign detection model and the traffic sign semantic segmentation model, thereby reducing the amount of calculation required to extract image features separately and improving the accuracy and timeliness of model processing. The sign coordinates and foreground segmentation results are input into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign, thereby realizing the complementarity between the prediction results of the traffic sign detection model and the traffic sign semantic segmentation model, and further improving the detection results of traffic sign detection.

[0071] Step 208, obtaining the distance between the traffic sign and the vehicle according to the target sign coordinates and the vehicle coordinates.

[0072] The method of this step has been described in the aforementioned step 104 and will not be repeated here.

[0073] For example, the sign information may include a sign type and a sign content, and the sign type may be a warning type, a reminder type, or other types. For example, the sign content may be pay attention to pedestrians, pay attention to children, pay attention to construction, etc.

[0074] Step 209: According to the sign type and the correspondence between the preset sign type and the preset distance judgment condition, the distance judgment condition corresponding to the sign type is obtained.

[0075] For example, the sign type includes a prompt type and a warning type. The sign type has a corresponding preset distance judgment condition, and the warning type has a corresponding preset distance judgment condition.

[0076] Step 210: If the distance between the traffic sign and the vehicle satisfies the distance judgment condition corresponding to the sign type, the sign content is output.

[0077] Furthermore, when the distance between the traffic sign and the vehicle does not satisfy the distance judgment condition corresponding to the sign type, the sign content is not output.

[0078] Specifically, the distance judgment condition includes a longitudinal distance judgment condition and a lateral distance judgment condition. If the lateral distance between the traffic sign and the vehicle satisfies the lateral distance judgment condition, and the longitudinal distance between the traffic sign and the vehicle satisfies the longitudinal distance judgment condition, it is determined that the distance between the traffic sign and the vehicle satisfies the distance judgment condition corresponding to the sign type; if the lateral distance between the traffic sign and the vehicle does not satisfy the lateral distance judgment condition, or the longitudinal distance between the traffic sign and the vehicle does not satisfy the longitudinal distance judgment condition, it is determined that the distance between the traffic sign and the vehicle satisfies the distance judgment condition corresponding to the sign type.

[0079] In the field of intelligent driving, vehicles can obtain traffic signs on the road through visual detection methods during driving. In order to ensure the efficiency and accuracy of subsequent processing based on traffic signs, it is necessary to ensure that the vehicle can detect traffic signs in real time and accurately in a variety of driving scenarios, and filter out traffic signs that are related to the vehicle and that the vehicle driver needs to pay attention to.

[0080] In this embodiment, according to the sign type and the correspondence between the preset sign type and the preset distance judgment condition, the distance judgment condition corresponding to the sign type is obtained, and the sign information is output when the distance satisfies the distance judgment condition corresponding to the sign type. In this way, traffic signs related to the vehicle and that the vehicle driver needs to pay attention to can be screened out. For different types of traffic signs, the corresponding distance judgment condition is used to determine whether to output the traffic sign, and the accuracy of screening the output traffic signs is improved through classification processing.

[0081] For example, when the sign type of the traffic sign is a warning type, step 209 may include the following sub-steps:

[0082] Sub-step 2091, determining whether there is a traffic light ahead of the road on which the vehicle is traveling.

[0083] For example, an image of the area in front of the road where the vehicle is traveling can be collected, and the area image can be input into a traffic light recognition model, and based on the recognition result output by the traffic light recognition model, it is determined whether there is a traffic light in front of the road where the vehicle is traveling. For example, the traffic light recognition model can be obtained by training an object detection model (for example, a YOLO model) using image samples with traffic lights as labels.

[0084] Sub-step 2092, if there is no traffic light ahead of the road on which the vehicle is traveling, then according to the sign type and the correspondence between the preset sign type and the preset distance judgment condition, obtain the distance judgment condition corresponding to the sign type.

[0085] For example, the correspondence relationship includes a first correspondence relationship and a second correspondence relationship; when the sign type of the traffic sign is a prompt type; step 209 may include the following sub-steps:

[0086] Sub-step 2093, determining whether the lane has a U-turn sign or a right turn sign.

[0087] For example, a road surface image of a vehicle driving on the road can be collected, and the road surface image can be input into a road surface sign recognition model to determine whether the lane has a U-turn sign or a right turn sign. For example, the road surface sign recognition model can be obtained by training an object detection model (for example, a YOLO model) using road surface image samples with U-turn signs or right turn signs as labels.

[0088] Sub-step 2094, if the lane has a U-turn sign or a right turn sign, then according to a first correspondence between a preset sign type and a preset distance judgment condition, a first distance judgment condition corresponding to the sign type is obtained.

[0089] For example, the first distance judgment condition includes: the absolute values ​​of the longitudinal distance and the lateral distance between the traffic sign and the vehicle are respectively within a first preset longitudinal distance range and a first preset lateral distance range.

[0090] Sub-step 2095, if the lane does not have a U-turn sign or a right turn sign, then according to a second correspondence between a preset sign type and a preset distance judgment condition, a second distance judgment condition corresponding to the sign type is obtained.

[0091] For example, the second distance judgment condition includes: the absolute values ​​of the longitudinal distance and the lateral distance between the traffic sign and the vehicle are respectively within a second preset longitudinal distance range and a second preset lateral distance range.

[0092] The second preset longitudinal distance range and the second preset lateral distance range may be 20 meters and 5 meters, respectively.

[0093] The first distance judgment condition includes a first sub-distance judgment condition and a second sub-distance judgment condition. If the lane has a U-turn sign or a right turn sign, after step 2093, the following further includes:

[0094] Sub-step 2095, obtaining the driving status of the vehicle.

[0095] The driving state includes going straight or turning.

[0096] The driving state of the vehicle can be determined according to the steering wheel angle. Specifically, the steering wheel angle of the vehicle is obtained, and when the steering wheel angle is greater than or equal to 90°, the driving state of the vehicle is determined to be a turning state, and when the steering wheel angle is less than 90°, the driving state of the vehicle is determined to be a straight state.

[0097] Furthermore, if the vehicle is traveling in a straight line state, step 210 may include the following sub-steps:

[0098] Sub-step 2101: if the distance between the traffic sign and the vehicle satisfies the first sub-distance judgment condition, the lane has a U-turn sign, and the U-turn sign is located on the left side of the vehicle, then the sign content of the traffic sign is output.

[0099] Among them, when the U-turn sign is located on the left side of the vehicle, it means that the lateral distance between the vehicle and the traffic sign is greater than zero.

[0100] The first sub-distance judgment condition includes: the absolute values ​​of the longitudinal distance and the lateral distance between the traffic sign and the vehicle are respectively within the first sub-preset longitudinal distance range and the first sub-preset lateral distance range. The first sub-preset longitudinal distance range and the first sub-preset lateral distance range can be 20 meters and 10 meters respectively.

[0101] Sub-step 2102: if the distance between the traffic sign and the vehicle satisfies the first sub-distance judgment condition, the lane has a right turn sign, and the right turn sign is located on the right side of the vehicle, then the sign content of the traffic sign is output.

[0102] If the driving state of the vehicle is turning, step 210 may include:

[0103] Sub-step 2103: if the distance between the traffic sign and the vehicle satisfies the second sub-distance judgment condition, the sign content of the traffic sign is directly output.

[0104] The second sub-distance judgment condition includes: the absolute values ​​of the longitudinal distance and the lateral distance between the traffic sign and the vehicle are respectively within the second sub-preset longitudinal distance range and the second sub-preset lateral distance range. The second sub-preset longitudinal distance range and the second sub-preset lateral distance range may be 20 meters and 8 meters respectively.

[0105] For example, if the sign type is a warning type, the sign content is any one of the first preset sign contents, and the sign content includes, step 210 may include the following sub-steps:

[0106] Sub-step 2104, if the distance between the traffic sign and the vehicle satisfies the third distance judgment condition, then obtaining the traffic light information ahead of the road on which the vehicle is traveling.

[0107] For example, the first preset sign content may include "watch out for pedestrians" and "watch out for children".

[0108] For example, the third distance judgment condition includes: the absolute values ​​of the longitudinal distance and the lateral distance between the traffic sign and the vehicle are within a third preset longitudinal distance range. The third preset longitudinal distance range is 20 meters.

[0109] For example, an image of the area in front of the road where the vehicle is traveling may be collected, and the image of the area may be input into a traffic light information acquisition model to determine the traffic light information in front of the road where the vehicle is traveling. The traffic light information in front of the road where the vehicle is traveling may include: a red light is on, a green light is on, or a yellow light is on.

[0110] For example, a traffic light recognition model can be a recognition model obtained by training a target detection model (for example, a YOLO model) using image samples of traffic lights of different colors as labels.

[0111] Sub-step 2105: if the traffic signal information is that the red light is on or the yellow light is on, then the sign content of the traffic sign is output.

[0112] If the sign type is a warning type and the sign content is any one of the second preset sign contents, step 210 may include the following sub-steps:

[0113] Sub-step 2106: if the distance between the traffic sign and the vehicle satisfies the third distance judgment condition, directly output the sign content of the traffic sign.

[0114] For example, the second preset sign content may include "Attention Construction".

[0115] That is, if the traffic sign is a construction warning sign, then when the distance between the traffic sign and the vehicle satisfies the third distance determination condition, the sign content of the traffic sign is directly output.

[0116] The following takes the example that the sign types include prompt type or warning type, where the prompt type includes signs such as stop and give way, slow down and give way, and the warning type includes signs such as pay attention to pedestrians, pay attention to children, and pay attention to construction, to further illustrate the traffic sign detection method of the present application.

[0117] Reference Figure 5 and Figure 6 The traffic signal detection method of the embodiment of the present application may include the following steps:

[0118] Step S1, obtaining vehicle coordinates and vehicle driving area images.

[0119] The method of this step has been described in the aforementioned step 201 and will not be repeated here.

[0120] Step S2, inputting the vehicle driving area image into a deep residual network to obtain a first image feature, and inputting the first image feature into a feature pyramid network to obtain a target image feature;

[0121] The deep residual network is the ResNet-101 network, and the feature pyramid network is the FPN network. The vehicle driving area image is input into the ResNet-101 network and the FPN network in turn to obtain the target image features.

[0122] Step S3: Input the target image features into the traffic sign detection model to obtain the traffic sign information and bounding box regression refinement results.

[0123] Specifically, the traffic sign target detection model of the present application is based on the classic Faster R-CNN algorithm, and the following improvements are made on this basis: ResNet-101 combined with the feature pyramid network FPN is used in the feature extraction network for feature extraction to obtain more detailed information, and feature extraction based on ResNet-101 combined with the feature pyramid network FPN can improve the detection results of small targets or targets in the distance in the intelligent driving scenario, thereby enabling the vehicle intelligent driving system to respond in advance according to the detected information and improve safety; the use of ROI Align for pooling operations reduces quantization errors, providing further guarantees for the output of accurate and reliable detection results.

[0124] Step S4, inputting the target image features into the traffic sign semantic segmentation model to obtain the pixel-level foreground segmentation result of the traffic sign.

[0125] Reference Figure 2 In an embodiment of the present application, the traffic sign detection network and the traffic sign semantic segmentation network are parallel and share the same feature extraction network, thereby achieving the purpose of saving computing resources.

[0126] Furthermore, the traffic sign semantic segmentation network uses the lightweight Mini-DeepLab module as the main part for processing, ensuring timeliness and accuracy.

[0127] Step S5, inputting the bounding box regression refinement result and the pixel-level foreground segmentation result into a multi-layer vector machine to obtain a corrected bounding box regression refinement result.

[0128] The model of this embodiment is an end-to-end large model that combines a deep neural network with a pixel-level traffic sign semantic segmentation module and a traffic sign detection model. It can combine the advantages of the two methods of target detection and semantic segmentation, and can meet the real-time and accuracy requirements of intelligent driving vehicles in the process of traffic sign detection. That is, through this embodiment, the real-time and accuracy of traffic sign detection can be ensured while saving computing resources, and accurate detection results can be obtained. By quickly screening the detected traffic signs based on the constraints obtained from experience, the intelligent driving system can do follow-up work for different scenarios, such as controlling the operation of the vehicle based on the screened traffic signs, and presenting the screened traffic signs to the driver for the driver's reference.

[0129] In this embodiment, the detection results of the two branches can be fused and refined by an information fusion and refinement module based on a multi-layer perceptron MLP, so as to achieve better detection and segmentation effects.

[0130] Step S6, converting the coordinate information in the bounding box regression refinement result into the target marker coordinates in the vehicle coordinate system through Zhang's calibration method.

[0131] Add restrictions to the detection results output by the traffic sign detection model, extract the effective information that needs to be considered in the current scenario, and pass the information to the downstream for subsequent operations to improve efficiency.

[0132] Step S7, if the sign type is a prompt type, determine whether there is a traffic light ahead of the road where the vehicle is traveling, if so, proceed to step S8, otherwise proceed to step S9.

[0133] Reference Figure 5 If the sign type is a prompt type, it is determined whether there is a traffic light ahead of the vehicle's road. If there is a traffic light ahead of the vehicle's road, it is determined that the traffic light status of the lane is invalid. Then, based on whether there are auxiliary signs on the road and the vehicle's driving status, it is determined whether a traffic sign needs to be output.

[0134] Step S8: Do not output the traffic sign content.

[0135] If the sign type is a warning type and there is no traffic light ahead of the vehicle, no traffic sign will be output.

[0136] Step S9, determining whether the lane where the vehicle is traveling has a U-turn sign or a right turn sign, if yes, proceed to step S10, otherwise proceed to step S12.

[0137] For example, based on the road surface image of the lane, it is determined whether the lane has a U-turn sign or a right turn sign.

[0138] Step S10: If the vehicle is traveling in a straight line state and the distance between the traffic sign and the vehicle satisfies the following first sub-distance judgment condition: the absolute value of the longitudinal distance does not exceed 20 meters, the lateral distance does not exceed 10 meters, and the U-turn sign is located on the left side of the vehicle, or the right turn sign is located on the right side of the vehicle, the sign content of the traffic sign is output.

[0139] Among them, the U-turn sign is located on the left side of the vehicle, indicating that the lateral distance between the vehicle and the traffic sign is greater than zero.

[0140] Reference Figure 5 If there is no traffic light ahead of the vehicle, it is determined that the traffic light ahead of the vehicle is invalid. If the traffic light is invalid, and the vehicle is currently in a straight state, there is a turning scene ahead of the lane, the absolute value of the longitudinal distance between the traffic sign vehicles does not exceed 20 meters, the lateral distance does not exceed 10 meters, the U-turn sign is on the left side of the vehicle, or the right turn sign is on the right side of the vehicle, output the traffic sign.

[0141] Step S11, if the driving state of the vehicle is turning, and the distance between the traffic sign and the vehicle meets the following second sub-distance judgment condition: the absolute value of the longitudinal distance does not exceed 20 meters, and the lateral distance does not exceed 8 meters, output the sign content of the traffic sign.

[0142] Reference Figure 5 , when the traffic light in front of the vehicle is invalid, the vehicle is about to turn, the absolute value of the longitudinal distance between the traffic sign and the vehicle does not exceed 20 meters, the lateral distance does not exceed 8 meters, and the absolute value of the steering wheel angle of the vehicle is greater than 90° (the vehicle is in a turning state), the traffic sign is output.

[0143] Step S12: If the distance between the traffic sign and the vehicle meets the following second distance judgment condition: the absolute value of the longitudinal distance does not exceed 20 meters, and the lateral distance does not exceed 5 meters, the sign content of the traffic sign is directly output.

[0144] Reference Figure 5 The traffic sign is a warning sign. There are no turning or U-turn signs on the road for vehicles. The absolute value of the longitudinal distance between the traffic sign and the vehicle does not exceed 20 meters, and the lateral distance does not exceed 5 meters. The sign type of the traffic sign is directly output.

[0145] Step S13, if the type information of the traffic sign is a warning type, determine whether the distance between the traffic sign and the vehicle satisfies the following third distance determination condition: the absolute value of the longitudinal distance does not exceed 20 meters, if so, proceed to step S14 or step S15, otherwise return to step S8.

[0146] Reference Figure 5 If the traffic sign is a warning sign and the longitudinal distance between the traffic sign and the vehicle is within 20 meters, determine whether the traffic sign is a sign to pay attention to pedestrians, children, or construction.

[0147] Step S14: if the traffic sign content is a pedestrian watch out sign or a child watch out sign, and the red light or yellow light ahead of the vehicle is on, then the traffic sign content is output.

[0148] Reference Figure 5 If the longitudinal distance between the traffic sign and the vehicle is within 20 meters, the type information of the traffic sign is a pedestrian or child sign, and the red or yellow light is on ahead of the vehicle, the traffic sign is output.

[0149] Step S15: if the content of the traffic sign is a construction warning sign, the content of the traffic sign is directly output.

[0150] Reference Figure 5 If the longitudinal distance between the traffic sign and the vehicle is within 20 meters and the type information of the traffic sign is a construction sign, the traffic sign is directly output.

[0151] In this embodiment, reference Figure 7 , a traffic sign detection device provided in an embodiment of the present application, the device 30 comprises:

[0152] The first acquisition module 301 is used to acquire the vehicle coordinates and the target image features of the vehicle driving area image;

[0153] The second acquisition module 302 is used to input the target image features into the traffic sign detection model to obtain the sign information and sign coordinates of the traffic sign in the vehicle driving area image, and input the target image features into the traffic sign semantic segmentation model to obtain the foreground segmentation result of the vehicle driving area image;

[0154] The third acquisition module 303 is used to input the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign;

[0155] The fourth acquisition module 304 is used to obtain the distance between the traffic sign and the vehicle according to the target sign coordinates and the vehicle coordinates, and output the sign information of the traffic sign when the distance between the traffic sign and the vehicle meets the distance judgment condition.

[0156] For example, the flag information includes flag content and flag type, and the fourth acquisition module 304 includes:

[0157] A first acquisition submodule, configured to acquire a distance judgment condition corresponding to a sign type according to the sign type and a correspondence between a preset sign type and a preset distance judgment condition;

[0158] The first output submodule is used to output the sign content in the sign type if the distance between the traffic sign and the vehicle meets the distance judgment condition corresponding to the sign type.

[0159] Optionally, the corresponding relationship includes a first corresponding relationship and a second corresponding relationship. When the sign type of the traffic sign is a prompt type, the first acquisition submodule includes:

[0160] The first judgment submodule is used to judge whether the lane where the vehicle is traveling has a U-turn sign or a right turn sign;

[0161] A second acquisition submodule is used to acquire a first distance judgment condition corresponding to the sign type according to a first correspondence between a preset sign type and a preset distance judgment condition if the lane has a U-turn sign or a right turn sign;

[0162] The third acquisition submodule is used to acquire a second distance judgment condition corresponding to the sign type according to a second corresponding relationship between a preset sign type and a preset distance judgment condition if there is no U-turn sign or a right turn sign in the lane.

[0163] Optionally, the first distance judgment condition includes a first sub-distance judgment condition and a second sub-distance judgment condition. If the lane has a U-turn sign or a right turn sign, the first acquisition submodule further includes:

[0164] The fourth acquisition submodule is used to acquire the driving state of the vehicle; the driving state includes straight driving or turning;

[0165] If the driving state of the vehicle is straight, the fourth acquisition module 304 includes:

[0166] A second output submodule is configured to output the content of the traffic sign if the distance between the traffic sign and the vehicle satisfies the first sub-distance judgment condition, the lane has a U-turn sign, and the U-turn sign is located on the left side of the vehicle; or,

[0167] The third output submodule is used to output the sign content of the traffic sign if the distance between the traffic sign and the vehicle meets the first sub-distance judgment condition, the lane has a right turn sign, and the right turn sign is located on the right side of the vehicle.

[0168] If the driving state of the vehicle is turning, the fourth acquisition module 304 includes:

[0169] The fourth output submodule is used to directly output the sign content of the traffic sign if the distance between the traffic sign and the vehicle meets the second sub-distance judgment condition.

[0170] Optionally, when the sign type of the traffic sign is a prompt type, the first acquisition submodule may include:

[0171] The second judgment submodule is used to judge whether there is a traffic light ahead of the road where the vehicle is traveling;

[0172] The fifth acquisition submodule is used to acquire the distance judgment condition corresponding to the sign type according to the sign type and the correspondence between the preset sign type and the preset distance judgment condition if there is no traffic light ahead of the road on which the vehicle is traveling.

[0173] Optionally, if the sign type is a warning type and the sign content is any one of the first preset sign contents, the fourth acquisition module 304 includes:

[0174] a sixth acquisition submodule, configured to acquire traffic light information ahead of the road on which the vehicle is traveling if the distance between the traffic sign and the vehicle satisfies a third distance determination condition;

[0175] a fifth output submodule, for outputting the sign content of the traffic sign if the traffic signal light information is that the red light is on or the yellow light is on;

[0176] If the sign type is a warning type and the sign content is any one of the second preset sign contents, the fourth acquisition module 304 includes:

[0177] The sixth output submodule is used to directly output the sign content of the traffic sign if the distance between the traffic sign and the vehicle meets the third distance judgment condition.

[0178] Optionally, the first acquisition module 301 includes:

[0179] A seventh acquisition submodule is used to acquire an image of the vehicle driving area;

[0180] An eighth acquisition submodule, used to input the image into a deep residual network to obtain a first image feature;

[0181] A ninth acquisition submodule, used for inputting the first image feature into a feature pyramid network to obtain a target image feature;

[0182] The second acquisition module 302 includes:

[0183] The tenth acquisition submodule is used to input the target image features into the region proposal network to obtain the proposal box prediction result;

[0184] The eleventh acquisition submodule is used to input the target image features and the suggestion box prediction results into the region of interest alignment network to obtain the sign type and sign coordinates of the traffic sign in the image.

[0185] In this embodiment, after obtaining the vehicle target image feature, the feature is input into the traffic sign detection model and the traffic sign semantic segmentation model, which can save computing resources, improve processing efficiency, and ensure the timeliness and accuracy of traffic sign detection. Based on the multi-layer perceptron, the detection results of the traffic sign detection model and the traffic sign semantic segmentation model are fused and processed, and accurate detection and segmentation effects can be obtained, and then accurate target sign coordinates can be obtained. According to the target sign coordinates and the vehicle coordinates, the distance between the traffic sign and the vehicle is obtained. When the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign information of the traffic sign is output, which is equivalent to adding restriction conditions to the detection results output by the traffic sign target detection model, judging whether the current scene needs to consider the sign information, and passing the information to the downstream for subsequent operations. In this way, the sign information that is not related to the vehicle can be filtered out, and the sign information related to the vehicle can be screened out, which improves the effectiveness of the detected sign information and improves the processing efficiency of subsequent processing based on the output sign type.

[0186] Reference Figure 8 , the electronic device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0187] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0188] The memory 404 is used to store various types of data to support the operation of the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, multimedia, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0189] The power supply component 406 provides power to the various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0190] The multimedia component 408 includes an interface that provides an output interface between the electronic device 400 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0191] The audio component 410 is used to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0192] The input / output I / O interface 412 provides an interface between the processing component 402 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0193] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of components, such as the display and keypad of the electronic device 400, and the sensor assembly 414 can also detect the position change of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0194] The communication component 416 is used to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0195] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to implement a service request processing method provided in an embodiment of the present application.

[0196] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of an electronic device 400 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0197] Fig. 9is a block diagram of an electronic device 500 according to another embodiment of the present invention. For example, the electronic device 500 may be provided as a server. Fig. 9 The electronic device 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532, for storing instructions that can be executed by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform a service request processing method provided in an embodiment of the present application.

[0198] The electronic device 500 may also include a power supply component 526 configured to perform power management of the electronic device 500, a wired or wireless network interface 550 configured to connect the electronic device 500 to a network, and an input / output (I / O) interface 558. The electronic device 500 may operate based on an operating system stored in the memory 532, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0199] An embodiment of the present application also provides a computer program product, including a computer program, and a traffic sign detection method implemented when the computer program is executed by a processor.

[0200] An embodiment of the present application further provides a vehicle, comprising the traffic sign detection device as described in the above embodiment, or the electronic device as described in the above embodiment.

[0201] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0202] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A traffic sign detection method, characterized in that: include: Obtaining vehicle coordinates and target image features of the vehicle driving area image; Inputting the target image features into a traffic sign detection model to obtain the sign information and sign coordinates of the traffic signs in the vehicle driving area image, and inputting the target image features into a traffic sign semantic segmentation model to obtain a foreground segmentation result of the vehicle driving area image; Inputting the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign; The distance between the traffic sign and the vehicle is obtained according to the target sign coordinates and the vehicle coordinates, and the sign information of the traffic sign is output when the distance between the traffic sign and the vehicle meets a distance judgment condition.

2. The method according to claim 1, characterized in that: The sign information includes sign content and sign type. When the distance between the traffic sign and the vehicle satisfies the distance judgment condition, the sign information of the traffic sign is output, including: According to the sign type and the correspondence between the preset sign type and the preset distance judgment condition, obtaining the distance judgment condition corresponding to the sign type; If the distance between the traffic sign and the vehicle satisfies a distance judgment condition corresponding to the sign type, the sign content is output.

3. The method according to claim 2, characterized in that The corresponding relationship includes a first corresponding relationship and a second corresponding relationship. When the sign type of the traffic sign is a prompt type, obtaining the distance judgment condition corresponding to the sign type according to the sign type and the corresponding relationship between the preset sign type and the preset distance judgment condition includes: Determine whether the lane the vehicle is traveling in has a U-turn sign or a right turn sign; If the lane has a U-turn sign or a right turn sign, obtaining a first distance judgment condition corresponding to the sign type according to a first correspondence between a preset sign type and a preset distance judgment condition; If the lane has no U-turn sign and no right turn sign, a second distance judgment condition corresponding to the sign type is obtained according to a second corresponding relationship between a preset sign type and a preset distance judgment condition.

4. The method according to claim 3, characterized in that The first distance judgment condition includes a first sub-distance judgment condition and a second sub-distance judgment condition. If the lane has a U-turn sign or a right turn sign, the method further includes: Acquire the driving state of the vehicle; the driving state includes straight driving or turning; If the driving state of the vehicle is straight ahead, then when the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign information of the traffic sign is output, including: If the distance between the traffic sign and the vehicle satisfies a first sub-distance judgment condition, the lane has a U-turn sign, and the U-turn sign is located on the left side of the vehicle, outputting the sign content of the traffic sign; If the distance between the traffic sign and the vehicle satisfies a first sub-distance judgment condition, the lane has a right turn sign, and the right turn sign is located on the right side of the vehicle, then outputting the sign content of the traffic sign; If the driving state of the vehicle is turning, then in the case where the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign information of the traffic sign is output, including: If the distance satisfies the second sub-distance judgment condition, the sign content of the traffic sign is directly output.

5. The method according to any one of claims 3 or 4, characterized in that: In the case where the sign type of the traffic sign is a prompt type, obtaining the distance judgment condition corresponding to the sign type according to the sign type and the correspondence between the preset sign type and the preset distance judgment condition includes: Determine whether there is a traffic light ahead of the road on which the vehicle is traveling; If there is no traffic light ahead of the road on which the vehicle is traveling, the distance judgment condition corresponding to the sign type is obtained according to the sign type and the correspondence between the preset sign type and the preset distance judgment condition.

6. The method according to claim 2, characterized in that If the sign type is a warning type and the sign content is any one of the first preset sign contents, then when the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign information of the traffic sign is output, including: If the distance between the traffic sign and the vehicle satisfies a third distance determination condition, obtaining traffic light information ahead of the road on which the vehicle is traveling; If the traffic light information indicates that the red light is on or the yellow light is on, outputting the sign content of the traffic sign; If the sign type is a warning type and the sign content is any one of the second preset sign contents, then when the distance between the traffic sign and the vehicle meets the distance judgment condition, the sign content of the traffic sign is output, including: If the distance between the traffic sign and the vehicle satisfies a third distance determination condition, the sign content of the traffic sign is directly output.

7. The method according to claim 1, characterized in that Obtain target image features of the vehicle driving area image, including: Acquire an image of the vehicle driving area; Inputting the vehicle driving area image into a deep residual network to obtain a first image feature; Inputting the first image feature into a feature pyramid network to obtain the target image feature; Inputting the target image features into a traffic sign detection model to obtain the sign information and sign coordinates of the traffic signs in the vehicle driving area image, including: Input the target image features into a region proposal network to obtain a proposal box prediction result; The target image features and the suggestion box prediction results are input into a region of interest alignment network to obtain the sign information and sign coordinates of the traffic sign in the image.

8. A traffic sign detection device, characterized in that: include: A first acquisition module is used to acquire vehicle coordinates and target image features of an image of a vehicle driving area; The second acquisition module is used to input the target image features into a traffic sign detection model to obtain the sign information and sign coordinates of the traffic signs in the vehicle driving area image, and input the target image features into a traffic sign semantic segmentation model to obtain the foreground segmentation result of the vehicle driving area image; A third acquisition module is used to input the sign coordinates and the foreground segmentation result into a multi-layer perceptron to obtain the target sign coordinates of the traffic sign; The fourth acquisition module is used to obtain the distance between the traffic sign and the vehicle according to the target sign coordinates and the vehicle coordinates, and output the sign information of the traffic sign when the distance between the traffic sign and the vehicle meets the distance judgment condition.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A vehicle, characterized in that: It includes the traffic sign detection device as claimed in claim 8, or the electronic device as claimed in claim 9.

Images