A traffic sign recognition method, system, device and storage medium

Abstract

The application discloses a traffic sign recognition method and system, wherein the method comprises the following steps: acquiring a video to be recognized; inputting all frame images of the video to be recognized into a trained traffic sign recognition model to output a traffic sign recognition result; wherein the trained traffic sign recognition model is used for: performing feature extraction on all frame images of the video to be recognized to obtain feature maps of all frame images; extracting traffic sign candidate regions from the feature maps of all frame images; performing feature fusion on the traffic sign candidate regions of a key frame image and the traffic sign candidate regions of adjacent frame images to obtain fused traffic sign candidate regions; performing feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps; and performing classification and regression on the fused candidate region feature maps to obtain the traffic sign recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a traffic sign recognition method, system, device and storage medium. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] With the rapid development of artificial intelligence and computer technology, machine vision is an important prerequisite for realizing unmanned driving, which means how to let the car "see" the world in front of it. Traffic sign recognition is an important field in unmanned driving. Traffic sign recognition of unmanned vehicles refers to obtaining road scene images using vehicle-mounted cameras and recognizing road signs and semantics on the images. It is an important content for unmanned vehicles to judge the current road indication. How to let the car automatically and accurately recognize the road sign has important research significance. In image processing, image data quality is one of the key influencing factors. However, in the acquisition process, the vehicle-mounted camera will inevitably encounter different degrees of occlusion under the interference of extreme weather or irresistible factors, and the quality of the collected images often fails to meet the requirements, which causes many inconveniences for subsequent data analysis.

[0004] Video target detection has certain advantages over static image target detection. Since the images in the video are continuous, there is an obvious context relationship between adjacent frames in the video. Therefore, when the target cannot be accurately tracked through a certain frame of the video, other frames with time, space and other context relationships with the current frame can be used to assist target detection of the current frame.

[0005] Chinese patent document CN116259032A discloses a road traffic sign detection and recognition algorithm based on improved YOLOv5, which realizes bidirectional fusion of deep and shallow features from top to bottom and from bottom to top, and significantly improves the detection performance of the network model. However, it cannot deal with traffic sign recognition in complex scenes.

[0006] Chinese patent document CN116152777A discloses a common traffic sign recognition method, system and storage medium based on YOLOv5. The present application not only has fast detection speed and high detection accuracy, but also occupies small video memory space and recognizes multiple target types. However, the traffic sign recognition rate under partial occlusion of the vehicle-mounted camera of the unmanned vehicle is low, and it cannot fully cope with this complex scene.

[0007] The patent CN113076842B discloses a method for improving the recognition accuracy of traffic signs in extreme weather and environment. Based on the YOLOv5 target detection model, the focusing module, cross-stage local fusion module and spatial pyramid pooling structure are integrated. For traffic sign images with poor light, the feature map information can be better extracted from local features, and the feature map more accurately represents the image. This patent does not solve the problem of traffic sign recognition under occlusion.

[0008] The vehicle-mounted camera is affected by a small part of occlusion (raindrops, mud, frost, snow, etc.) in extreme weather, which affects the recognition accuracy of traffic signs in target detection, especially the recognition accuracy of small target traffic signs, leading to frequent errors such as misjudgment of unmanned vehicles. Therefore, the small part of occlusion (raindrops, mud, frost, snow, etc.) of the vehicle-mounted camera of the unmanned vehicle interferes with the feature information extraction of the target detection model, making it difficult for the model to obtain positive and effective semantic information, resulting in low recognition accuracy. SUMMARY

[0009] To solve the problems of the prior art, the present application provides a traffic sign recognition method, system, device and storage medium. The technical problem of the prior art that the target recognition of traffic signs under small part of occlusion of the vehicle-mounted camera cannot accurately identify the target is solved.

[0010] In one aspect, a traffic sign recognition method is provided.

[0011] A traffic sign recognition method includes:

[0012] Obtaining a video to be recognized;

[0013] Dividing all frame images of the video to be recognized into key frame images and adjacent frame images; the key frame image refers to a frame with a different scene from the previous frame image but the same scene as the next frame image; the adjacent frame image refers to the image between the current key frame image and the previous key frame image, and the image between the current key frame image and the next key frame image;

[0014] Inputting all frame images of the video to be recognized into a trained traffic sign recognition model, and outputting a traffic sign recognition result; wherein the trained traffic sign recognition model is used for: performing feature extraction on all frame images of the video to be recognized to obtain feature maps of all frame images; extracting traffic sign candidate regions from the feature maps of all frame images; performing feature fusion on the traffic sign candidate regions of the key frame images and the traffic sign candidate regions of the adjacent frame images to obtain fused traffic sign candidate regions; performing feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps; and performing classification and regression on the fused candidate region feature maps to obtain a traffic sign recognition result.

[0015] In another aspect, a traffic sign recognition system is provided.

[0016] A traffic sign recognition system comprises:

[0017] An acquisition module configured to acquire a video to be recognized;

[0018] A key frame division module configured to divide all frame images of the video to be recognized into key frame images and adjacent frame images; the key frame image refers to a frame that is different from a scene of a previous frame image but the same as a scene of a next frame image; the adjacent frame image refers to an image between a current key frame image and a previous key frame image and an image between the current key frame image and a next key frame image;

[0019] A traffic sign recognition module configured to input all frame images of the video to be recognized into a trained traffic sign recognition model and output a traffic sign recognition result; wherein the trained traffic sign recognition model is configured to: perform feature extraction on all frame images of the video to be recognized to obtain feature maps of all frame images; extract traffic sign candidate regions from the feature maps of all frame images; perform feature fusion on the traffic sign candidate regions of the key frame images and the traffic sign candidate regions of the adjacent frame images to obtain fused traffic sign candidate regions; perform feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps; and perform classification and regression on the fused candidate region feature maps to obtain the traffic sign recognition result.

[0020] In another aspect, an electronic device is also provided, comprising:

[0021] a memory for non-transiently storing computer readable instructions; and

[0022] a processor for running the computer readable instructions,

[0023] wherein the computer readable instructions, when run by the processor, perform the method of the first aspect.

[0024] In another aspect, a storage medium is also provided, which non-transiently stores computer readable instructions, wherein the non-transient computer readable instructions, when executed by a computer, perform the instructions of the method of the first aspect.

[0025] In another aspect, a computer program product is also provided, comprising a computer program for implementing the method of the first aspect when run on one or more processors.

[0026] One of the above technical solutions has the following advantages or beneficial effects:

[0027] The key frame in the video stream is determined from the pre-acquired video; then from the time span of adjacent key frames, the context information correlation is found, the RS loss function is introduced instead of the original classification loss function, and the problem of traffic sign class imbalance is solved. In order to better compensate the semantic information of the occluded target image, the graph convolutional neural network is introduced based on the correlation coefficient matrix, and the model recognition rate is improved.

[0028] The application introduces a similarity-based attention mechanism in the network structure, constructs a context feature memory, supplements the missing semantic information, and improves the recognition effect under occlusion. The application introduces the RS Loss loss function instead of the original classification loss function, solves the problem of traffic sign class imbalance. The graph convolutional neural network is introduced in the network structure of the application, based on the correlation coefficient matrix, the semantic information of the occluded target image is compensated, and the model recognition rate is improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] The drawings constituting a part of the specification of the application are used to provide further understanding of the application, the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute improper limitation on the application.

[0030] Figure 1 The overall flowchart of the embodiment one of the application is shown in the figure;

[0031] Figure 2 The similarity-based attention module of the embodiment one of the application is shown in the figure;

[0032] Figure 3 The overall detection flow of the embodiment one of the application is shown in the figure;

[0033] Figure 4(a) is a standard convolutional layer of the embodiment one of the application;

[0034] Figure 4(b) is a deformable convolutional layer of the embodiment one of the application;

[0035] Figure 5 The overall network structure of the embodiment one of the application is shown in the figure;

[0036] Figure 6 The Fused-MBConv convolutional structure of the embodiment one of the application is shown in the figure. DETAILED DESCRIPTION

[0037] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.

[0038] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0039] Embodiment one

[0040] The embodiment provides a traffic sign recognition method;

[0041] As shown in Figure 1 and Figure 3 , a traffic sign recognition method comprises:

[0042] S101: acquiring a video to be recognized;

[0043] S102: dividing all frame images of the video to be recognized into key frame images and adjacent frame images; the key frame image refers to a frame whose scene is different from that of a previous frame image but the same as that of a subsequent frame image; the adjacent frame image refers to an image between a current key frame image and a previous key frame image and an image between the current key frame image and a subsequent key frame image;

[0044] S103: inputting all frame images of the video to be recognized into a trained traffic sign recognition model to output a traffic sign recognition result; wherein the trained traffic sign recognition model is used for:

[0045] performing feature extraction on all frame images of the video to be recognized to obtain feature maps of all frame images;

[0046] extracting traffic sign candidate regions from the feature maps of all frame images;

[0047] performing feature fusion on the traffic sign candidate regions of the key frame images and the traffic sign candidate regions of the adjacent frame images to obtain fused traffic sign candidate regions;

[0048] performing feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps;

[0049] performing classification and regression on the fused candidate region feature maps to obtain a traffic sign recognition result.

[0050] Further, in step S101: acquire the video to be identified by using an in-vehicle camera to capture the traffic sign image.

[0051] Furthermore, such as Figure 5 As shown, step S103 involves inputting all frame images of the video to be recognized into the trained traffic sign recognition model and outputting the traffic sign recognition result; wherein, the network structure of the trained traffic sign recognition model includes:

[0052] The backbone network and the candidate region generation network are connected sequentially;

[0053] The output of the candidate region generation network is connected to the input of the similarity attention module through a first branch, and the first branch is provided with a region of interest pooling layer; the input of the region of interest pooling layer RoI Pooling is connected to the output of the candidate region generation network, and the output of the region of interest pooling layer RoI Pooling is connected to the input of the similarity attention module.

[0054] The output of the candidate region generation network is connected to the input of the similarity attention module through a second branch;

[0055] The output of the similarity attention module is connected to the input of the graph convolutional neural network;

[0056] The output of the convolutional neural network is connected to the input of the first fully connected layer.

[0057] The output of the first fully connected layer is connected to the regressor and the classifier, respectively.

[0058] Furthermore, such as Figure 6 As shown, the backbone network includes: a first 3*3 convolutional layer, an SE layer, a first 1*1 convolutional layer, an adder, an activation function layer, and a pooling layer connected in sequence;

[0059] The input of the 3x3 convolutional layer is also connected to the input of the adder.

[0060] The first 3x3 convolutional layer is implemented using deformable convolution.

[0061] The Squeeze-and-Excitation (SE) module is used to enhance the feature representation capabilities of convolutional neural networks (CNNs). The SE module compresses the feature map of each channel into a scalar using global average pooling to obtain global statistical information for each channel. Then, it generates a channel weight vector through two fully connected layers (FC layers) to weight the feature map of each channel.

[0062] It should be understood that the present embodiment adopts EfficientNetV2 as the backbone feature extraction network. The core module of EfficientNetV2 adopts a Fused-MBConv convolution, as shown in Figure 6 The 3x3 depth convolution and the 1x1 convolution in the MBConv are replaced by a conventional 3x3 convolution, as shown in Figure 5 It has less parameter quantity and calculation quantity. The convolution kernel of the convolution layer in EfficientNet is fixed size and size, and the fixed convolution kernel has poor feature extraction capability for morphological change targets. In view of the characteristics of multi-scale traffic signs and the difficulty in extracting features, deformable convolution is introduced into the feature extraction network to enhance the adaptability of the network. The improved backbone feature extraction network replaces part of the standard convolution with deformable convolution to enhance the extraction capability of the network for multi-scale traffic signs.

[0063] As shown in FIGS. 4(a) and 4(b), the deformable convolution additionally adds a convolution layer to the input feature map to learn the offset of each sampling point, and then adds the learned offset to the original convolution kernel to make the standard convolution window become an offset window, and then performs a conventional convolution operation on the input feature map. The deformable convolution enhances the extraction capability of the network for small target traffic signs.

[0064] The present application introduces deformable convolution into the backbone network, which adds a convolution layer to the standard convolution. The input of the deformable convolution is a feature map, and the offset of each sampling point is learned, and then the learned offset is added to the standard convolution to make the standard convolution window become an offset window, and then the deformable convolution operation is performed on the input feature map. The deformable convolution disclosed in the present application effectively enhances the extraction capability of the network for small target traffic signs.

[0065] Further, as shown in Figure 5 The candidate region generation network comprises:

[0066] a second 3*3 convolution layer, a second 1*1 convolution layer, a first Reshape layer, an activation function layer, a second Reshape layer and a Proposal layer connected in sequence;

[0067] The output end of the second 3*3 convolution layer is further connected to the input end of the second Reshape layer through a third 1*1 convolution layer.

[0068] It should be understood that the Reshape layer is a commonly used layer in neural networks, and its main function is to reshape the input tensor according to the specified shape. The number of elements of the input and output tensors remains unchanged, but the shape can be changed.

[0069] It should be understood that the Proposal layer, which mainly generates a series of bounding boxes according to the input feature map, provides the subsequent target classification and position regression network for use.

[0070] Further, as shown in Figure 2 The similar attention module comprises:

[0071] The three parallel sub-branches are a first sub-branch, a second sub-branch and a third sub-branch.

[0072] The first sub-branch comprises a pooling layer P1, a second full connection layer and a normalization layer G1 connected in sequence.

[0073] The second sub-branch comprises a third full connection layer and a normalization layer G2 connected in sequence.

[0074] The third sub-branch comprises a fourth full connection layer.

[0075] An input end of the pooling layer P1 is connected with an output end of the region of interest pooling layer.

[0076] An input end of the third full connection layer and an input end of the fourth full connection layer are both connected with an output end of the candidate region generation network.

[0077] An output end of the normalization layer G1 and an output end of the normalization layer G2 are both connected with an input end of an activation function layer S1, and an output end of the activation function layer S1 is connected with an input end of a fifth full connection layer.

[0078] An output end of the fourth full connection layer is connected with an input end of the fifth full connection layer.

[0079] An output end of the fifth full connection layer is an output end of the similar attention module.

[0080] Further, as shown in Figure 5 The S103 comprises: inputting all frame images of a to-be-identified video into a trained traffic sign recognition model to output a traffic sign recognition result.

[0081] A training set is constructed, and the training set is a traffic sign video with known traffic sign recognition results.

[0082] The training set is input into the traffic sign recognition model to train the model, and when a loss function value of the model no longer decreases, or when an iteration number exceeds a set number of times, the training is stopped to obtain the trained traffic sign recognition model.

[0083] Further, the feature extraction on all frame images of the to-be-identified video to obtain feature maps of the frame images comprises:

[0084] Based on the backbone network, feature extraction is performed on all frame images of the to-be-identified video to obtain feature maps of all frame images.

[0085] Further, the feature maps of all frame images are extracted to obtain traffic sign candidate boxes and traffic sign candidate regions, including:

[0086] Based on the candidate region generation network, the feature maps of all frame images are extracted to obtain traffic sign candidate boxes and traffic sign candidate regions.

[0087] Further, the traffic sign candidate regions of the key frame image and the traffic sign candidate regions of the adjacent frame image are fused to obtain the fused traffic sign candidate regions, including:

[0088] Based on the similar attention module, the traffic sign candidate regions of the key frame image and the traffic sign candidate regions of the adjacent frame image are fused to obtain the fused traffic sign candidate regions.

[0089] Further, the fused traffic sign candidate regions are subjected to feature map extraction to generate a fused candidate region feature map, including:

[0090] Based on the graph convolutional neural network, the fused traffic sign candidate regions are subjected to feature map extraction to generate a fused candidate region feature map.

[0091] Further, the fused candidate region feature map and the traffic sign candidate region of the key frame are classified and regressed to obtain a traffic sign recognition result, including:

[0092] Based on the regressor and the classifier, the fused candidate region feature map and the traffic sign candidate region of the key frame are classified and regressed to obtain a traffic sign recognition result.

[0093] The steps of the present example mainly include the following contents: first, the position range of the traffic sign target in the current video frame image is obtained through the vehicle-mounted camera to determine the key frame x in the video; wherein the key frame refers to the scene of the current frame being different from the scene of the previous frame, and being the same as the scene of the next frame; a context storage space is constructed from the time span of two adjacent key frames;

[0094] First, the key frame is set; the key frame in the video sequence is selected as the first frame key frame, and a plurality of adjacent frames are determined from the context storage space of the specified frame corresponding time span.

[0095] Because the traffic sign image captured by the vehicle-mounted camera probe is partially blocked, some key semantic information is missing, so in order to obtain better detection effect, a GCN module is added in the basic network to extract the feature map. In the GCN module, the feature map is converted into a graph structure, and a graph convolution network is used for feature extraction and graph analysis.

[0096] In addition, when a specific task is imbalanced during the multi-task training process, such as the imbalance of positive and negative samples in the classification task, additional hyperparameters are generated, and the fine-tuning of these hyperparameters occupies computing time and can lead to suboptimal results. Therefore, to solve the problem of unbalanced data samples, the classification loss function is optimized in this embodiment, and the RS Loss (Rank & Sort Loss) loss function is introduced instead of the original classification loss function, solving the problem of unbalanced traffic sign categories and improving the model recognition rate.

[0097] When the vehicle-mounted camera captures a video stream under partial occlusion, the target detection is performed on the obtained video stream. Through analysis, it is concluded that since the target is in a stationary state and the vehicle-mounted camera is in a constant moving state, the problem of low recognition rate caused by occlusion of the vehicle-mounted camera is solved by using the context relationship to complete the feature and the graph convolution neural network feature prediction method to improve the detection accuracy and facilitate better analysis and judgment of the unmanned vehicle.

[0098] In addition, when a specific task is imbalanced during the multi-task training process, such as the imbalance of positive and negative samples in the classification task, additional hyperparameters are generated, and the fine-tuning of these hyperparameters occupies computing time and can lead to suboptimal results. Therefore, to solve the problem of unbalanced data samples, the classification loss function is optimized in this embodiment, and the RS Loss loss function is introduced instead of the original classification loss function, solving the problem of unbalanced traffic sign categories and improving the model recognition rate. The loss function is composed of classification loss and bounding box regression loss. When a specific task is imbalanced during the multi-task training process, such as the imbalance of positive and negative samples in the classification task, additional hyperparameters are generated, and the fine-tuning of these hyperparameters occupies computing time and can lead to suboptimal results. To solve the problem of unbalanced data samples, this paper introduces Rank & Sort Loss (RS Loss) in the loss function. RS Loss arranges positive samples above negative samples when calculating loss, and sorts positive samples according to the size of IoU value. Based on the sorting characteristics, RS Loss can handle unbalanced data and simplify the training model. This paper uses RS Loss function instead of the original classification loss function of Faster R-CNN. Solving the problem of unbalanced traffic sign categories and improving the model recognition rate.

[0099] The key frames in the video stream are determined from the pre-acquired video. Then, from the time span of adjacent key frames, the contextual information correlation is found, and a context memory M is constructed. The deformable convolution is used to replace part of the standard convolution to enhance the feature extraction capability. The RS Loss is introduced to replace the original classification loss function to solve the problem of traffic sign class imbalance and improve the model recognition rate. A multi-label classification model based on a graph convolutional network (GCN) is introduced to learn the mutual dependence between labels, thereby improving the accuracy of region proposal and the performance of the entire target detection model, thereby solving the problem of low traffic sign recognition rate under partial occlusion of the vehicle-mounted camera of the unmanned vehicle.

[0100] Embodiment Two

[0101] The embodiment provides a traffic sign recognition system.

[0102] The traffic sign recognition system comprises:

[0103] The acquisition module is configured to acquire a video to be recognized.

[0104] The key frame division module is configured to divide all frame images of the video to be recognized into key frame images and adjacent frame images. The key frame image refers to a frame whose scene is different from that of the previous frame image but the same as that of the next frame image. The adjacent frame image refers to an image between the current key frame image and the previous key frame image and an image between the current key frame image and the next key frame image.

[0105] The traffic sign recognition module is configured to input all frame images of the video to be recognized into a trained traffic sign recognition model and output a traffic sign recognition result. The trained traffic sign recognition model is used to perform feature extraction on all frame images of the video to be recognized to obtain feature maps of all frame images, extract traffic sign candidate regions from the feature maps of all frame images, perform feature fusion on the traffic sign candidate regions of the key frame images and the traffic sign candidate regions of the adjacent frame images to obtain fused traffic sign candidate regions, perform feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps, and perform classification and regression on the fused candidate region feature maps to obtain the traffic sign recognition result.

[0106] It should be noted that the above acquisition module, key frame division module and traffic sign recognition module correspond to steps S101 to S103 in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules can be executed in a computer system such as a set of computer executable instructions as part of a system.

[0107] The description of each embodiment in the above embodiments has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0108] The proposed system can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the above modules is only a logical function division, and in actual implementation, there can be another division method, for example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0109] Embodiment Three

[0110] The embodiment also provides an electronic device, including one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected with the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in Embodiment One.

[0111] It should be understood that in the embodiment, the processor can be a central processing unit CPU, and the processor can also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, ready-to-program gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0112] The memory can include read-only memory and random access memory, and provide instructions and data for the processor, and a part of the memory can also include non-volatile random access memory. For example, the memory can also store device type information.

[0113] In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software.

[0114] The method in the embodiment one can be directly embodied as being completed by a hardware processor, or being completed by a combination of hardware and software modules in the processor. The software modules can be located in a storage medium in the art, such as a random access memory, a flash memory, a read only memory, a programmable read only memory, an electrically erasable programmable memory, a register, or the like. The storage medium is located in the storage, and the processor reads information in the storage, and combines the hardware to complete the steps of the above method. To avoid repetition, no further detailed description is given here.

[0115] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0116] Embodiment four

[0117] The embodiment also provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in embodiment one.

[0118] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A traffic sign recognition method, characterized by, The method comprises the following steps: acquire a video to be identified; divide all frame images of the video to be identified into key frame images and adjacent frame images; the key frame image refers to a frame image whose scene is different from that of the previous frame image but the same as that of the next frame image; the adjacent frame image refers to an image between the current key frame image and the previous key frame image and an image between the current key frame image and the next key frame image; input all frame images of the video to be identified into a trained traffic sign recognition model to output a traffic sign recognition result; wherein the trained traffic sign recognition model is used to: perform feature extraction on all frame images of the video to be identified to obtain feature maps of all frame images; extract traffic sign candidate regions from the feature maps of all frame images; perform feature fusion on the traffic sign candidate regions of the key frame images and the traffic sign candidate regions of the adjacent frame images to obtain fused traffic sign candidate regions; perform feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps; and perform classification and regression on the fused candidate region feature maps to obtain the traffic sign recognition result; the network structure of the trained traffic sign recognition model comprises: a backbone network and a candidate region generation network connected in sequence; the backbone network comprises a first 3*3 convolutional layer, an SE layer, a first 1*1 convolutional layer, an adder, an activation function layer and a pooling layer connected in sequence; the input end of the 3*3 convolutional layer is also connected with the input end of the adder; the first 3*3 convolutional layer is realized by using deformable convolution; the output end of the candidate region generation network is connected with the input end of a similar attention module through a first branch, and a region of interest pooling layer is arranged on the first branch; the input end of the region of interest pooling layer RoI Pooling is connected with the output end of the candidate region generation network, and the output end of the region of interest pooling layer RoI Pooling is connected with the input end of the similar attention module; the output end of the candidate region generation network is connected with the input end of the similar attention module through a second branch; the output end of the similar attention module is connected with the input end of a graph convolutional neural network; the output end of the graph convolutional neural network is connected with the input end of a first full connection layer; the output end of the first full connection layer is connected with a regressor and a classifier respectively.

2. The traffic sign recognition method of claim 1, wherein the step of identifying the traffic sign comprises the steps of: determining a distance between the traffic sign and the vehicle; and identifying the traffic sign based on the determined distance. the candidate region generation network comprises a second 3*3 convolutional layer, a second 1*1 convolutional layer, a first reshape layer, an activation function layer, a second reshape layer and a proposal layer connected in sequence; the output end of the second 3*3 convolutional layer is also connected with the input end of the second reshape layer through a third 1*1 convolutional layer.

3. The traffic sign recognition method as described in claim 1, characterized in that, the similar attention module comprises three parallel sub-branches: a first sub-branch, a second sub-branch and a third sub-branch; the first sub-branch comprises a pooling layer P1, a second full connection layer and a normalization layer G1 connected in sequence; the second sub-branch comprises a third full connection layer and a normalization layer G2 connected in sequence; and the third sub-branch comprises a fourth full connection layer. An input end of the pooling layer P1 is connected with an output end of the region of interest pooling layer; An input end of the third full connection layer and an input end of the fourth full connection layer are both connected with an output end of the candidate region generation network; An output end of the normalization layer G1 and an output end of the normalization layer G2 are both connected with an input end of the activation function layer S1; an output end of the activation function layer S1 is connected with an input end of the fifth full connection layer; An output end of the fourth full connection layer is connected with an input end of the fifth full connection layer; An output end of the fifth full connection layer is an output end of the similar attention module.

4. The traffic sign recognition method as described in claim 1, characterized in that, All frame images of the video to be recognized are input into the trained traffic sign recognition model, and a traffic sign recognition result is output; wherein the training process of the trained traffic sign recognition model includes: A training set is constructed, and the training set is a traffic sign video with known traffic sign recognition results; The training set is input into the traffic sign recognition model to train the model; when the loss function value of the model no longer decreases, or when the iteration number exceeds the set number of times, the training is stopped, and the trained traffic sign recognition model is obtained.

5. The traffic sign recognition method of claim 1, wherein the step of identifying the traffic sign comprises the steps of: determining a distance between the traffic sign and the vehicle; and identifying the traffic sign based on the determined distance. The feature extraction of all frame images of the video to be recognized to obtain the feature map of all frame images includes: ​ Based on the backbone network, the feature extraction of all frame images of the video to be recognized is performed to obtain the feature map of all frame images; The feature map of all frame images is extracted to obtain the traffic sign candidate frame and the traffic sign candidate region, including: Based on the candidate region generation network, the feature map of all frame images is extracted to obtain the traffic sign candidate frame and the traffic sign candidate region; The traffic sign candidate region of the key frame image and the traffic sign candidate region of the adjacent frame image are fused based on the similar attention module to obtain the fused traffic sign candidate region; The feature map extraction of the fused traffic sign candidate region to generate the fused candidate region feature map includes: Based on the graph convolutional neural network, the feature map extraction of the fused traffic sign candidate region is performed to generate the fused candidate region feature map; The classification and regression of the fused candidate region feature map and the traffic sign candidate region of the key frame are performed to obtain the traffic sign recognition result, including: Based on the regressor and the classifier, the classification and regression of the fused candidate region feature map and the traffic sign candidate region of the key frame are performed to obtain the traffic sign recognition result. It includes:

6. A traffic sign recognition system, characterized in that An acquisition module configured to acquire a video to be recognized; A key frame division module configured to divide all frame images of the video to be recognized into key frame images and adjacent frame images; the key frame image refers to a frame with a scene different from that of the previous frame but the same as that of the next frame; the adjacent frame image refers to an image between the current key frame image and the previous key frame image, and an image between the current key frame image and the next key frame image; ​ The traffic sign recognition module is configured to input all frame images of a to-be-recognized video into a trained traffic sign recognition model, and output a traffic sign recognition result; wherein the trained traffic sign recognition model is configured to: perform feature extraction on all frame images of the to-be-recognized video to obtain feature maps of all frame images; extract traffic sign candidate regions from the feature maps of all frame images; perform feature fusion on the traffic sign candidate regions of a key frame image and the traffic sign candidate regions of adjacent frame images to obtain fused traffic sign candidate regions; perform feature map extraction on the fused traffic sign candidate regions to generate fused candidate region feature maps; and perform classification and regression on the fused candidate region feature maps to obtain the traffic sign recognition result. The network structure of the trained traffic sign recognition model comprises: a backbone network and a candidate region generation network connected in sequence; the backbone network comprises: a first 3*3 convolutional layer, an SE layer, a first 1*1 convolutional layer, an adder, an activation function layer and a pooling layer connected in sequence; the input end of the 3*3 convolutional layer is also connected to the input end of the adder; the first 3*3 convolutional layer is implemented by using deformable convolution; the output end of the candidate region generation network is connected to the input end of the similar attention module through a first branch, and a region of interest pooling layer is arranged on the first branch; the input end of the region of interest pooling layer RoI Pooling is connected to the output end of the candidate region generation network, and the output end of the region of interest pooling layer RoI Pooling is connected to the input end of the similar attention module; the output end of the candidate region generation network is connected to the input end of the similar attention module through a second branch; the output end of the similar attention module is connected to the input end of the graph convolutional neural network; the output end of the graph convolutional neural network is connected to the input end of the first full connection layer; the output end of the first full connection layer is connected to the regressor and the classifier respectively.

7. An electronic device, comprising: a memory for non-transiently storing computer readable instructions; and a processor for running the computer readable instructions, wherein the computer readable instructions are run by the processor to perform the method of any one of claims 1-5.

8. A storage medium characterized by, non-transiently storing computer readable instructions, wherein when the non-transient computer readable instructions are executed by a computer, the instructions perform the method of any one of claims 1-5.

Images