Multi-modal detection and identification system and method for automatic driving speed limit identifier

Abstract

The invention provides a multi-modal detection and recognition system and method for an automatic driving speed limit identifier, and relates to the technical field of automatic driving environment perception, and the method comprises the steps: obtaining an environment image, environment point cloud data and vehicle operation data; performing lightweight coarse detection by using the preprocessed environment image, inputting the environment image into an improved YOLOv8-tiny model, and outputting to obtain a speed limit identifier candidate area; performing multi-feature fine recognition based on the candidate region, inputting the digital region image into an STN-CRNN network, and outputting to obtain a digital visual feature; key point cloud region interception is carried out on the preprocessed environment point cloud data, a key point cloud set is clustered based on a density clustering algorithm to obtain a target cluster, and point cloud modal comprehensive features of the target cluster are extracted; identifying a current road section ID by using the vehicle operation data, and extracting map modal features of the current road section; and carrying out dynamic weight fusion on the digital visual features, the point cloud modal comprehensive features and the map modal features, and finally outputting speed limit information.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of automatic driving environment perception, in particular to an automatic driving speed limit sign multi-modal detection and recognition system and method. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] With the development of automatic driving technology to L3 and above, real-time and high-precision recognition of traffic signs by the environment perception system has become a core requirement to ensure driving safety. As a key basis for dynamically adjusting the speed of a vehicle, the recognition accuracy of speed limit signs directly affects the safety and compliance of automatic driving decisions. Vehicle speeding accidents caused by misidentification of speed limit signs account for a large proportion every year, highlighting the necessity of accurate detection and recognition of speed limit signs.

[0004] Currently, automatic driving environment perception mainly relies on multi-sensor fusion solutions such as vision, lidar, and millimeter wave radar. Among them, vision sensors, due to their low cost and rich information (including color, character, and other semantic features), have become the mainstream sensing means for speed limit sign recognition. However, complex road scenes (such as heavy rain, backlight, occlusion, and sign pollution) and non-standard signs (such as temporary construction signs and handwritten and altered signs) pose serious challenges to traditional vision recognition technology, and it is urgent to improve robustness through multi-modal fusion and intelligent decision-making.

[0005] In existing methods, traditional computer vision (such as template matching and color threshold segmentation) or pure deep learning (such as YOLO+OCR) are used to achieve sign detection and character recognition. However, in complex weather conditions (rain, fog, rain, and snow), visual features are blurred, recognition accuracy is reduced, and in backlight scenes, sign details are easily lost due to overexposure. Additionally, relying solely on two-dimensional images cannot distinguish between real signs and false targets such as reflections and stickers. Introducing lidar or millimeter wave radar to assist vision recognition, through point cloud data to obtain the three-dimensional size of the target, is mostly a serial strategy combining vision detection with radar filtering, without deeply fusing cross-modal features (such as the correlation between vision color and radar reflectivity), and the utilization rate of redundant information is low. Furthermore, the decision logic is not adjusted in combination with the vehicle's dynamic state (such as current speed and road type), and in high-speed scenarios, instantaneous misjudgment can easily lead to emergency braking risks. SUMMARY

[0006] The present disclosure proposes an automatic driving speed limit sign multi-modal detection and recognition system and method to solve the above problems. Through a two-stage detection and recognition mechanism, the potential speed limit sign area is identified through the first-stage coarse detection process, and the accurate point cloud features are obtained through the second-stage fine detection process. The image and point cloud multi-modal fusion obtains preliminary speed limit information, and the closed-loop verification is realized through real-time vehicle speed feedback, and the dynamic decision of the automatic driving speed limit sign is realized.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: An automatic driving speed limit sign multi-modal detection and recognition method comprises: acquiring environment images, environment point cloud data and vehicle operation data, and preprocessing them; Lightweight coarse detection is performed on the preprocessed environment images, the environment images are input into an improved YOLOv8-tiny model, and candidate regions of speed limit signs are output; Multi-feature fine recognition is performed based on the candidate regions, color features are extracted from the candidate regions, and circular character contour regions are extracted; the circular character contour regions are transformed, internal digital regions are extracted, digital region images are input into an STN-CRNN network, and digital visual features are output; Key point cloud regions are intercepted from the preprocessed environment point cloud data, key point cloud sets are clustered based on a density clustering algorithm, target clusters are obtained, and point cloud modal comprehensive features of the target clusters are extracted; the current road section ID is identified using the vehicle operation data, and map modal features of the current road section are extracted; The digital visual features, the point cloud modal comprehensive features and the map modal features are dynamically weighted and fused, and finally the speed limit information is output.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: An automatic driving speed limit sign multi-modal detection and recognition system comprises: A data acquisition module is configured to acquire environment images, environment point cloud data and vehicle operation data, and preprocess them; A coarse recognition module is configured to perform lightweight coarse detection on the preprocessed environment images, input the environment images into an improved YOLOv8-tiny model, and output candidate regions of speed limit signs; The fine identification module is used for multi-feature fine identification based on the candidate region, extracting a color feature from the candidate region, and extracting a circular character contour region; the circular character contour region is subjected to a transformation operation, an internal digital region is extracted, and a digital region image is input into an STN-CRNN network to output a digital visual feature; key point cloud regions are intercepted from the preprocessed environment point cloud data, the key point cloud sets are clustered based on a density clustering algorithm to obtain target clusters, and point cloud modal comprehensive features of the target clusters are extracted; a current road section ID is identified by using vehicle operation data, and a map modal feature of the current road section is extracted; the digital visual feature, the point cloud modal comprehensive feature, and the map modal feature are dynamically weighted and fused, and finally speed limit information is output.

[0009] According to some embodiments, the present disclosure adopts the technical scheme as follows: A computer program product comprising a computer program which, when executed by a processor, implements the automatic driving speed limit sign multi-modal detection and recognition method.

[0010] According to some embodiments, the present disclosure adopts the technical scheme as follows: A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the automatic driving speed limit sign multi-modal detection and recognition method.

[0011] According to some embodiments, the present disclosure adopts the technical scheme as follows: An electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the automatic driving speed limit sign multi-modal detection and recognition method.

[0012] Compared with the prior art, the present disclosure has the beneficial effects that: The automatic driving speed limit sign multi-modal detection and recognition method of the present disclosure fuses three types of modal data of visual images, laser radar point clouds, and high-precision maps (preset speed limit / road section type), realizes environment self-adaptation through dynamic weights (such as rain laser radar weight promotion to 0.6), and has an identification accuracy greater than that of a traditional single visual scheme in heavy rain, backlight, and complex shielding scenes, thereby solving the limitations of pure vision dependence on light and texture features.

[0013] The automatic driving speed limit sign multi-modal detection and recognition method of the present disclosure improves the detection recall rate of long-distance small-size speed limit signs and reduces the risk of speeding caused by missed detection by using an improved YOLOv8-tiny model for coarse detection in the first stage and combining color verification and character recognition for fine screening in the second stage.

[0014] The automatic driving speed limit sign multi-modal detection and recognition method of the present disclosure improves the robustness, real-time performance, and safety of traditional speed limit sign recognition schemes in complex environments, reduces the hardware cost and energy consumption of autonomous driving perception systems, and promotes the large-scale application of technology. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which form a part of this disclosure, are intended to provide further understanding of the present disclosure and are incorporated herein for illustrative purposes. The schematic embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0016] Figure 1 A flowchart of an automatic driving speed limit sign multi-modal detection and recognition method according to an embodiment of the present disclosure is shown in FIG. 1. Figure 2 An improved YOLOv8-tiny model architecture diagram according to an embodiment of the present disclosure is shown in FIG. 2. DETAILED DESCRIPTION

[0017] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0018] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0019] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component, and / or combinations thereof.

[0020] Embodiment 1 An automatic driving speed limit sign multi-modal detection and recognition method is provided in an embodiment of the present disclosure. The method steps include: Step 1: Obtain environmental images, environmental point cloud data, and vehicle operation data, and pre-process them; Step two: light rough detection is performed using the preprocessed environmental image. The environmental image is input into the improved YOLOv8-tiny model, and the speed limit sign candidate region is output. Step three: multi-feature fine identification is performed based on the candidate region. Color features are extracted from the candidate region, and a circular character contour region is extracted. The circular character contour region is transformed, and an internal number region is extracted. The number region image is input into the STN-CRNN network, and the number visual feature is output. Step four: key point cloud region interception is performed on the preprocessed environmental point cloud data. The key point cloud set is clustered based on the density clustering algorithm to obtain a target cluster, and the point cloud modal comprehensive feature of the target cluster is extracted. The current road section ID is identified using vehicle operation data, and the map modal feature of the current road section is extracted. Step five: the number visual feature, the point cloud modal comprehensive feature, and the map modal feature are dynamically weighted and fused to finally output the speed limit information.

[0021] As an embodiment, an automatic driving speed limit sign multi-modal detection and recognition method of the present disclosure is provided. Through a two-stage detection and recognition mechanism, potential speed limit sign regions are identified through a first-stage rough detection process, and then a second-stage fine detection process is performed to realize the recognition of the automatic driving speed limit sign, thereby improving safe driving. The specific implementation process is as follows: Step 1: Obtain environmental images, environmental point cloud data, and vehicle operation data, and pre-process them. Specifically, the CMOS camera is installed at the rearview mirror of the front windshield, with a resolution of 1920x1080 and a frame rate of 30fps. It collects RGB environmental images within a 120° field of view in front of the vehicle, and outputs data in YUV420 format. The laser radar is installed at the center of the roof, using a 128-line solid-state radar with a horizontal field of view of 120° and a vertical field of view of 30°, and a point cloud density of 200 points / m2. It outputs three-dimensional point cloud data. The IMU positioning module supports RTK differential positioning and outputs vehicle operation data, including real-time latitude and longitude (WGS84 coordinate system), heading angle (accuracy ±0.1°), and vehicle speed (error ≤0.5 km / h).

[0022] Through timestamp alignment (error ≤10ms), the environmental images, point cloud data, and vehicle operation (positioning) data are packaged into perception data frames.

[0023] Further, the obtained data is pre-processed, and the pre-processing process includes: (1) For environmental images: Distortion correction: Based on the camera intrinsic matrix and distortion coefficients, the original image is corrected for radial / tangential distortion. The camera intrinsic matrix is obtained by Zhang's calibration method.

[0024] Light normalization: The CLAHE algorithm is used to enhance the local contrast, control the image brightness variance in the range of [50, 200], and eliminate the feature loss caused by backlight / shadow.

[0025] ROI extraction: The road area is segmented by a semantic segmentation model, and the lower 60% area of the image (excluding sky, trees and other non-road area interference) is intercepted. The semantic segmentation model can be a lightweight SegNet.

[0026] (2) For environmental point cloud data: De-noising: Use voxel filtering (voxel size 0.05m x 0.05m x 0.05m) to remove discrete noise points.

[0027] Ground segmentation: The ground plane is fitted by the RANSAC algorithm, and its model is: ax+by+cz+d=0. The ground points are filtered out, and the target point cloud with a height of 0.5-3m (typical installation height of speed limit signs) is retained. Coordinate conversion: Convert the point cloud from the radar coordinate system to the camera coordinate system (rotate and translate through the external parameter matrix), and realize the spatial alignment of "image-point cloud".

[0028] Step 2: Use the pre-processed environment image for lightweight coarse detection. The environment image is input into the improved YOLOv8-tiny model, and the speed limit sign candidate region is output. Specifically, the improved YOLOv8-tiny model is improved based on YOLOv8-tiny. The model structure is improved based on the CSPDarknet skeleton and PANneck feature fusion layer of the basic YOLOv8-tiny model.

[0029] The improved YOLOv8-tiny model of the present disclosure splits the 3x3 standard convolution into 3x3 depth convolution DepthwiseConv and 1x1 pointwise convolution Pointwise Conv, and replaces the depth separable convolution with the standard convolution. The original 5 CSPBlock of CSPDarknet is reduced to 4, reducing the bottom redundant calculation, and retaining 3 effective feature output layers, including P3, P4 and P5, corresponding to 8x8, 16x16, 32x32 downsampling rate respectively.

[0030] Further, after the P3 (small target feature layer) and P4 (medium target feature layer) outputs of the PANneck feature fusion network, a new channel attention (Channel Attention) and spatial attention (Spatial Attention) serial module is added to enhance the color feature response of the speed limit sign. Among them, the channel attention input feature map F，After global average pooling GAP, global maximum pooling GMP and shared MLP layer full connection, the channel weight is output by using Sigmoid activation; then spatial attention is performed, and the result of element-by-element multiplication of the feature map and the channel and spatial weight is obtained.

[0031] In the detection head position, the YOLOv8-tiny detection head outputs classification / regression features through 3x3 convolution and 1x1 convolution, and contains 3 detection branches (category, confidence, coordinates). The category branch is simplified in the present disclosure, and only one category of "speed limit sign" is reserved (the original model is 80 categories of COCO general target), the number of convolution kernels of the classification head is reduced from 255 to 18, and a "decoupling head" structure is adopted, but the 3x3 convolution of the regression branch is replaced by 1x1 convolution, and a dynamic activation function DyReLU is used to replace the original SiLU activation function, so that the activation threshold is adaptively adjusted according to the input feature, and the response of small target features is enhanced.

[0032] As an embodiment, the preprocessed environment image is input into the improved YOLOv8-tiny model for coarse recognition. First, the pixel value is normalized and converted into a model input sequence format, and then feature extraction is performed in the improved YOLOv8-tiny model. Through convolution and down-sampling, P3 (52x52x256), P4 (26x26x512), and P5 (13x13x1024) feature maps are output. The P5 feature map fuses the P4 feature, and the P4 feature fuses the P3 feature to obtain the fused P3 feature. The fused P3 (52x52x256) and P4 (26x26x512) feature maps are respectively output through the CBAM module to obtain the attention weighted feature map. Each feature map grid generates 3 anchor boxes, and each anchor box outputs a candidate region of the speed limit sign through the coordinate decoding formula of YOLOv8.

[0033] Step 3: Multi-feature fine recognition based on the candidate region, color features are extracted from the candidate region, and a circular character contour region is extracted; the circular character contour region is transformed to extract an internal number region, and the number region image is input into the STN-CRNN network to output a number visual feature. Specifically, the color features are extracted from the candidate region, and the HSV space is: red H∈[0,10]∪[160,180], S∈[0.4,1.0], V∈[0.3,1.0]); and the shape of the contour is fitted (circularity ≥ 0.8, diameter 40-80 cm; The filtered circular region is perspective transformed (corrected for tilt), and the internal number region is intercepted and input into the STN-CRNN network. The spatial transformation layer STN corrects the character distortion, and outputs the number recognition result (such as "60" "80") and the character confidence (char_score). Step 4: Key point cloud region interception is performed on the preprocessed environment point cloud data, density-based clustering algorithm is used for clustering of the key point cloud set, target clusters are obtained, and point cloud modal comprehensive features of the target clusters are extracted; Specifically, first, according to the pixel coordinates (x1, y1, x2, y2) of the visual candidate frame, the four boundary line equations of the candidate frame in the image coordinate system are calculated through the camera intrinsic matrix K (fx, fy, cx, cy) and the distortion coefficient; in combination with the extrinsic matrix (rotation matrix R, translation vector t) of the laser radar and the camera, the image coordinate system boundary line is projected into the radar three-dimensional coordinate system to obtain a three-dimensional bounding box (ROI cube): From the preprocessed point cloud, the point cloud subset (denoted as Cloud candidate) located in the above three-dimensional bounding box is intercepted, irrelevant points (such as the ground and distant trees) outside the box are filtered out, and the calculation amount is reduced.

[0034] Based on the density-based clustering algorithm-DBSCAN algorithm, the Cloud candidate point cloud is input, the clustering parameters are set, and the output is: the clustered target cluster, and each cluster represents an independent three-dimensional target (such as speed limit sign, street lamp, traffic cone, etc.).

[0035] The number of points N of each cluster is calculated: if N < 20 (sparse point cloud, which may be noise) or N > 500 (dense point cloud, which may be a large object such as a billboard), the cluster is directly removed; The mean intensity of the reflectivity of the points in the cluster is calculated: the surface of the speed limit sign is high-reflectivity paint (reflectivity is usually > 0.6), and if the intensity_mean < 0.3 (such as dark tree trunk), the cluster is removed; Further, geometric feature extraction is performed on the target cluster, the three-dimensional coordinate extreme value of the target cluster in the radar coordinate system is calculated, the three-dimensional region corresponding to the candidate frame is intercepted in the aligned point cloud, the target size is calculated after clustering, and if the size matches the circular sign feature, the point cloud matching degree is output.

[0036] Step 5: Identify the current road segment ID using vehicle operation data, and extract the map modal features of the current road segment; The current road segment ID is obtained using the segmentation model, the standard speed limit value and the road segment type of the current road segment are obtained, and the map modal features are obtained; Step 6: Dynamically weighted fusion of digital visual features, point cloud modal comprehensive features, and map modal features, and finally output the speed limit information.

[0037] Specifically, the weights of each modality are dynamically adjusted according to the real-time environment, and environmental factors are introduced, including: light intensity: the ambient light value (unit: lux) is obtained through the camera sensor, and is divided into "strong light (>10000 lux)", "normal (1000-10000 lux)", and "weak light (<1000 lux)"; weather condition (weather_condition): obtained through camera image analysis (such as raindrop detection, fog concentration estimation) or vehicle-mounted meteorological sensor, divided into "sunny", "rainy", "foggy", "snowy"; vehicle speed (vehicle_speed): the current vehicle speed (unit: km / h) is obtained through the CAN bus, divided into "low speed (≤40)", "medium speed (40-80)", and "high speed (>80)".

[0038] According to the environmental factors, the fusion weights of vision (w_visual), lidar (w_lidar), and high-precision map (w_map) are adjusted to realize the fusion of multi-modal features, and finally the speed limit information is output.

[0039] According to the speed limit information, the vehicle speed is adjusted, and the current vehicle speed is uploaded to the server in real time to realize the corresponding feedback of the database road section and the vehicle speed, and to expand the database.

[0040] Embodiment 2 An embodiment of the present disclosure provides an automatic driving speed limit sign multi-modal detection and recognition system, comprising: A data acquisition module is configured to acquire environmental images, environmental point cloud data, and vehicle operation data, and pre-process the same. A coarse recognition module is configured to perform lightweight coarse detection on the pre-processed environmental images, input the environmental images into an improved YOLOv8-tiny model, and output a speed limit sign candidate region. A fine recognition module is configured to perform multi-feature fine recognition based on the candidate region, extract color features from the candidate region, and extract a circular character contour region; perform a transformation operation on the circular character contour region, extract an internal number region, input the number region image into an STN-CRNN network, and output a number visual feature; perform key point cloud region interception on the pre-processed environmental point cloud data, cluster the key point cloud set based on a density clustering algorithm, obtain a target cluster, and extract a point cloud modality comprehensive feature of the target cluster; identify a current road section ID using the vehicle operation data, and extract a map modality feature of the current road section; and dynamically weight and fuse the number visual feature, the point cloud modality comprehensive feature, and the map modality feature, and finally output speed limit information.

[0041] Embodiment 3 An embodiment of the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the automatic driving speed limit sign multi-modal detection and recognition method.

[0042] Embodiment 4 An embodiment of the present disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the automatic driving speed limit sign multi-modal detection and recognition method.

[0043] Embodiment 5 An embodiment of the present disclosure provides an electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the automatic driving speed limit sign multi-modal detection and recognition method.

[0044] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable data processing apparatus to generate a computer-implemented process, so that the instructions executed by the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0046] The above describes the specific embodiments of the present disclosure in conjunction with the accompanying drawings, but is not a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications or changes made to the technical solutions of the present disclosure without inventive labor are still within the protection scope of the present disclosure.

Claims

1. A multimodal detection and recognition method for speed limit signs in autonomous driving, characterized in that, include: Acquire environmental images, environmental point cloud data, and vehicle operation data, and preprocess them; Lightweight coarse detection is performed using preprocessed environmental images. The environmental images are input into the improved YOLOv8-tiny model, and the output is the candidate region for speed limit markers. Based on candidate regions, multi-feature fine recognition is performed, color features are extracted from candidate regions, and circular character outline regions are extracted. The circular character outline region is transformed to extract the internal digit region. The digit region image is input into the STN-CRNN network to output the digit visual features. The key point cloud regions are extracted from the preprocessed environmental point cloud data, and the key point cloud sets are clustered based on the density clustering algorithm to obtain target clusters. The point cloud modal comprehensive features of the target clusters are then extracted. Use vehicle operation data to identify the current road segment ID and extract the map modal features of the current road segment; The digital visual features, point cloud modal features, and map modal features are dynamically weighted and fused to ultimately output speed limit information.

2. The method for multimodal detection and recognition of speed limit signs for autonomous driving as described in claim 1, characterized in that, The system uses a CMOS camera mounted on the rearview mirror of the windshield to acquire RGB images of the front of the vehicle, a LiDAR mounted on the center of the roof to acquire 3D environmental point cloud data, an IMU module to acquire the vehicle's real-time position and speed, and timestamps to align the multimodal data. The system performs distortion correction, illumination normalization and POI extraction on the environmental images, and performs preprocessing operations such as noise reduction, ground segmentation and coordinate transformation on the environmental point cloud data.

3. The method for multimodal detection and recognition of speed limit signs for autonomous driving as described in claim 1, characterized in that, The improved YOLOv8-tiny model is based on the YOLOv8-tiny model. It replaces the standard convolution with depthwise separable convolution, splitting the 3×3 standard convolution into 3×3 depthwise convolution and 1×1 pointwise convolution. The original CSPDarknet's 5 CSPBlocks are reduced to 4, retaining 3 effective feature output layers. The dynamic activation function DyReLU replaces the original SiLU activation function. After the output of the small target feature layer P3 and the medium target feature layer P4 of the PANneck feature fusion network, a new channel attention and spatial attention serial module is added. In the detection head module, only one category is identified by speed. A decoupled head structure is adopted, but the 3×3 convolution in the regression branch is replaced with a 1×1 convolution.

4. The method for multimodal detection and recognition of speed limit signs for autonomous driving as described in claim 1, characterized in that, Color features are extracted from candidate regions, and circular contours are fitted to detect shapes. Perspective transformation is performed on the selected circular regions to extract the internal digit regions, which are then input into the STN-CRNN network. The spatial transformation layer STN is used to correct character distortion, and the digit recognition results and character confidence scores are output. The corresponding 3D regions of the candidate boxes are extracted from the aligned point cloud, and the target size is calculated after clustering. If the size matches the features of the circular sign, the point cloud matching degree is output.

5. The method for multimodal detection and recognition of speed limit signs for autonomous driving as described in claim 1, characterized in that, By using vehicle operation data to identify the current road segment ID, extracting the map modal features of the current road segment, and dynamically adjusting the weights of each modality according to the real-time environment, environmental factors, including light intensity, weather, and vehicle speed, are introduced. Based on these environmental factors, the fusion weights of vision, LiDAR, and high-precision maps are adjusted to achieve dynamic fusion.

6. The method for multimodal detection and recognition of speed limit signs for autonomous driving as described in claim 1, characterized in that, Based on the pixel coordinates of the candidate region for the speed limit sign, the equations of the four boundary lines of the candidate box in the image coordinate system are calculated using the camera intrinsic parameter matrix K and distortion coefficients. Combining the extrinsic parameter matrices of the LiDAR and the camera, the boundary lines of the image coordinate system are projected onto the radar's three-dimensional coordinate system to obtain a three-dimensional bounding box. From the preprocessed point cloud, a subset of the point cloud located within the above three-dimensional bounding box is extracted, and the DBSCAN algorithm is used to cluster the point cloud to obtain the clustered target clusters. Each cluster represents an independent three-dimensional target. The geometric features of the target clusters are extracted and spatially aligned with the candidate region for the speed limit sign, and the point cloud modal comprehensive features are output.

7. A multimodal detection and recognition system for speed limit signs in autonomous driving, characterized in that, include: The data acquisition module is used to acquire environmental images, environmental point cloud data, and vehicle operation data, and to preprocess them; The coarse recognition module is used to perform lightweight coarse detection using the preprocessed environmental image. The environmental image is input into the improved YOLOv8-tiny model, and the output is the candidate region of the speed limit label. The fine recognition module is used for multi-feature fine recognition based on candidate regions, extracting color features from candidate regions, and extracting circular character outline regions; The circular character outline region is transformed to extract the internal digit region. The digit region image is input into the STN-CRNN network to output the digit visual features. The key point cloud regions are extracted from the preprocessed environmental point cloud data, and the key point cloud sets are clustered based on the density clustering algorithm to obtain target clusters. The point cloud modal comprehensive features of the target clusters are then extracted. Use vehicle operation data to identify the current road segment ID and extract the map modal features of the current road segment; The digital visual features, point cloud modal features, and map modal features are dynamically weighted and fused to ultimately output speed limit information.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal detection and recognition method for speed limit signs in autonomous driving as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the multimodal detection and recognition method for speed limit signs in autonomous driving as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a multimodal detection and recognition method for speed limit signs for autonomous driving as described in any one of claims 1-6.

Images