Traffic sign recognition method, apparatus, device, and medium

Abstract

The application discloses a traffic sign recognition method, device, equipment and medium, the method comprises: traffic sign body detection is carried out to the traffic image to be detected, and the traffic body area image corresponding to the traffic image to be detected is obtained;Determine N anchor boxes in the traffic body area image, detect the traffic sign in each anchor box corresponding area, and obtain the traffic sign area image corresponding to the traffic sign body area image;The sign type detection is carried out to the traffic sign area image, and the type information of the traffic sign in the traffic image to be detected is obtained, and the traffic sign recognition result of the traffic image to be detected is obtained based on the type information.The technical scheme can more fine-grainedly extract the image features of each traffic sign area in the traffic body area image, determines the traffic sign recognition result of the traffic image to be detected by carrying out the sign type detection to the traffic sign area image, combines more fine and comprehensive features, and improves the accuracy of the traffic sign recognition result.

Description

Technical Field

[0001] This invention generally relates to the field of intelligent transportation technology, and specifically to a traffic sign recognition method, device, equipment, and medium. Background Technology

[0002] With the continuous development of science and technology, intelligent transportation technologies such as autonomous driving, assisted driving, and intelligent navigation are increasingly being applied to users' daily lives. In the application of intelligent transportation technologies, the recognition of traffic lights and other traffic signs is particularly important for accurately navigating users and ensuring the safety of autonomous and assisted driving.

[0003] Currently, related technologies can input images captured during driving into a classification model. The classification model first detects the location of traffic lights in the image, then crops the detection box region, and finally classifies and recognizes the detection box region to obtain the traffic light recognition result.

[0004] However, traffic lights typically have multiple lights on a single unit, and the semantics conveyed when multiple lights are illuminated are often quite complex. Currently, it's impossible to exhaustively define the semantics of traffic lights. Training models based on limited semantics results in poor model performance and an inability to recognize the multi-attribute semantics of traffic lights in practical applications, leading to low accuracy in traffic light recognition. Summary of the Invention

[0005] In view of the aforementioned defects or deficiencies in the existing technology, it is desirable to provide a traffic sign recognition method, apparatus, device, and medium that can extract image features of each traffic sign region in a traffic sign body image with finer granularity, thereby improving the accuracy of traffic sign recognition results for the traffic image to be detected. The technical solution is as follows:

[0006] According to one aspect of this application, a traffic sign recognition method is provided, the method comprising:

[0007] Traffic sign detection is performed on the traffic image to be detected to obtain the traffic sign region image corresponding to the traffic image to be detected.

[0008] N anchor frames are determined in the traffic body region image. Traffic sign detection is performed on the region corresponding to each anchor frame to obtain the traffic sign region image corresponding to the traffic sign body region image. The size and position of each of the N anchor frames are different, and N is an integer greater than or equal to 1.

[0009] The traffic sign area image is subjected to sign type detection to obtain the type information of traffic signs in the traffic image to be detected, and the traffic sign recognition result of the traffic image to be detected is obtained based on the type information.

[0010] According to another aspect of this application, a traffic sign recognition device is provided, the device comprising:

[0011] The traffic body detection module is used to detect traffic signs in the traffic image to be detected, and to obtain the traffic body region image corresponding to the traffic image to be detected.

[0012] The sign region detection module is used to determine N anchor frames in the traffic body region image, and to perform traffic sign detection on the region corresponding to each anchor frame to obtain the traffic sign region image corresponding to the traffic sign body region image; the size and position of each of the N anchor frames are different, and N is an integer greater than or equal to 1;

[0013] The recognition module is used to perform sign type detection on the traffic sign area image, obtain the type information of traffic signs in the traffic image to be detected, and obtain the traffic sign recognition result of the traffic image to be detected based on the type information.

[0014] According to another aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the traffic sign recognition method as described above.

[0015] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon for implementing the traffic sign recognition method as described above.

[0016] According to another aspect of this application, a computer program product is provided, which includes instructions that, when executed, implement the traffic sign recognition method as described above.

[0017] The traffic sign recognition method, apparatus, device, and medium provided in this application embodiment detect traffic signs in a traffic image to be detected, obtaining a traffic sign region image corresponding to the traffic image to be detected. Then, N anchor frames are determined in the traffic sign region image, and traffic sign detection is performed on the region corresponding to each anchor frame, obtaining a traffic sign region image corresponding to the traffic sign body region image. Furthermore, sign type detection is performed on the traffic sign region image to obtain the type information of the traffic signs in the traffic image to be detected. Based on the type information, the traffic sign recognition result of the traffic image to be detected is obtained. Compared with the prior art, the technical solution in this application, on the one hand, by determining N anchor frames in the traffic sign region image after obtaining the guiding information of the traffic sign region image, allows for more precise traffic sign detection on the region corresponding to each anchor frame, extracting image features of each traffic sign region in the traffic sign region image with finer granularity. This enables the recognition of traffic signs in the image based on more detailed features, effectively improving the accuracy of traffic sign recognition. On the other hand, by performing sign type detection on the traffic sign area image, and combining more refined and comprehensive features to determine the traffic sign recognition result of the traffic image to be detected, the recognition accuracy of the method provided in this application can be significantly improved compared with the prior art to a certain extent.

[0018] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0020] Figure 1 A schematic diagram showing the comparison structure between the enhanced sample and the real sample in the related technology provided in the embodiments of this application;

[0021] Figure 2 System architecture diagram of the traffic sign recognition application system provided in the embodiments of this application;

[0022] Figure 3 A flowchart illustrating the traffic sign recognition method provided in this application embodiment;

[0023] Figure 4 This is a schematic diagram of the structure of traffic sign recognition provided in an embodiment of this application;

[0024] Figure 5 A flowchart illustrating the method for obtaining traffic sign areas provided in an embodiment of this application;

[0025] Figure 6This is a schematic diagram of the structure for obtaining output parameters through a detection network, provided in an embodiment of this application.

[0026] Figure 7 This is a schematic diagram of the internal structure of a detection network provided in another embodiment of this application;

[0027] Figure 8 A schematic diagram of the structure for preprocessing traffic sign area images provided in an embodiment of this application;

[0028] Figure 9 A flowchart illustrating the training and detection network method provided in this application embodiment;

[0029] Figure 10 This is a flowchart illustrating a method for obtaining traffic sign recognition results provided in an embodiment of this application.

[0030] Figure 11 This is a schematic diagram of the structure for preprocessing traffic sign area images provided in an embodiment of this application.

[0031] Figure 12 This is a schematic diagram of the internal structure of the classification network provided in an embodiment of this application;

[0032] Figure 13 A flowchart illustrating the traffic sign recognition method provided in this application embodiment;

[0033] Figure 14 This is a schematic diagram of the traffic sign recognition method provided in the embodiments of this application;

[0034] Figure 15 This is a structural diagram of the area for obtaining traffic signs provided in an embodiment of this application;

[0035] Figure 16 This is a schematic diagram of the structure for obtaining traffic sign recognition results provided in an embodiment of this application;

[0036] Figure 17 This is a schematic diagram of the traffic sign recognition device provided in the embodiments of this application;

[0037] Figure 18 This is a schematic diagram of the traffic sign recognition device provided in the embodiments of this application;

[0038] Figure 19 This is a schematic diagram of the structure of a computer device as shown in an embodiment of this application. Detailed Implementation

[0039] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments. For ease of understanding, some technical terms involved in the embodiments of this application are explained below:

[0041] (1) Artificial Intelligence (AI): AI refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software primarily includes computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] (2) Machine Learning (ML): This is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0044] (3) Convolutional Neural Network (CNN): A feedforward neural network that includes convolutional computation and has a deep structure, it is one of the representative algorithms of deep learning. CNNs have the ability to learn representations and can perform translation-invariant classification of input information according to their hierarchical structure.

[0045] (4) Traffic signs: These are road facilities that use words or symbols to convey guidance, restriction, warning, or instruction information. They are also called road signs or road traffic signs. Traffic signs come in many types and can be distinguished in various ways: main signs and auxiliary signs; movable signs and fixed signs; illuminated signs, luminous signs, and reflective signs; and variable information signs that reflect changes in the driving environment, such as speed limit signs and traffic lights on the road.

[0046] (5) Traffic lights: These are traffic lights that direct traffic. They generally consist of three lights: red, green, and yellow. Red indicates no entry, green indicates entry, and yellow indicates a warning. They can be categorized as: motor vehicle traffic lights, non-motor vehicle traffic lights, pedestrian crossing traffic lights, directional indicator lights, lane traffic lights, flashing warning traffic lights, and traffic lights at road and railway level crossings.

[0047] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0048] The solutions provided in this application involve technologies such as neural networks in artificial intelligence, which are specifically illustrated in the following embodiments.

[0049] Currently, related technologies employ a two-stage approach: first, the location of traffic lights is detected; then, a detection bounding box is cropped, and the cropped region is classified to obtain the traffic light recognition result. Each type of traffic light is considered a category. However, this method, by treating each type of light as a single category, results in only one semantic meaning per light. When multiple lights on a single light are illuminated (multi-lighting), the multi-attribute semantics of the light cannot be recognized. Furthermore, if multi-lighting lights are defined as a separate category—for example, when a light has both a left-turn red light and a straight-ahead red light illuminated simultaneously—the category can be defined as "left-turn red light and straight-ahead red light." This results in a large number of categories arising from different shapes, colors, quantities, and types of traffic lights, making it impossible to enumerate all possible categories when locating the classification result. This leads to poor model performance and low accuracy in traffic light recognition. Additionally, due to the limited number of rare light category samples, augmentation processing is required when building the dataset. However, the results of augmented samples do not accurately reflect the actual sample situation. (See [link to previous section]). Figure 1 As shown, for example, a vertical right-turn red light can be obtained by flipping a vertical left-turn red light. However, for a horizontal right-turn red light, the enhanced samples obtained by enhancement (flipping or rotating) do not match the real horizontal right-turn red light, resulting in an insufficient number of horizontal right-turn samples, which in turn makes the trained model have low recognition accuracy.

[0050] To address the aforementioned deficiencies, this application provides a traffic sign recognition method, apparatus, device, and medium. Compared with existing technologies, after obtaining the guiding information of a traffic body area image, traffic sign detection is performed, thereby enabling more granular extraction of image features for each traffic sign area in the traffic body area image. Furthermore, by performing sign type detection on the traffic sign area image, more refined and comprehensive features are combined to determine the traffic sign recognition result of the traffic image to be detected, thus improving the accuracy of the traffic sign recognition result.

[0051] Figure 2 This is an implementation environment architecture diagram of a traffic sign recognition method provided in an embodiment of this application. For example... Figure 2 As shown, the implementation environment architecture includes: terminal 10 and server 20.

[0052] In the field of image recognition, the process of recognizing traffic signs in a traffic image to be detected can be performed either on terminal 10 or on server 20. For example, by acquiring a traffic image to be detected through terminal 10, image recognition can be performed locally on terminal 10 to obtain the recognition result of the traffic sign to be recognized; alternatively, the traffic image to be detected can be sent to server 20, so that server 20 can acquire the traffic image to be detected, perform image recognition based on the traffic image to obtain the recognition result of the traffic sign to be recognized, and then send the recognition result of the traffic sign to be recognized to server 20, thereby realizing the recognition of the type of traffic sign to be recognized in the traffic image to be detected.

[0053] The traffic sign recognition scheme provided in this application can be applied to common scenarios such as autonomous driving, vehicle navigation, map data collection, road data collection, intelligent transportation, and assisted driving. In these application scenarios, it is usually necessary to collect real-world road images, analyze them to obtain information such as traffic sign recognition results, and then perform subsequent operations based on this information, such as map updates, route planning, and controlling autonomous driving.

[0054] In addition, the terminal 10 may run an operating system, which may include, but is not limited to, Android, iOS, Linux, Unix, Windows, etc. It may also include a user interface (UI) layer, which can provide the display of real-time traffic images and the display of the recognition results of traffic signs to be recognized. Furthermore, it can send the real-time traffic images required for image recognition to the server 20 based on the application programming interface (API).

[0055] Optionally, terminal 10 can be a terminal device in various AI application scenarios. For example, terminal 10 can be a laptop, tablet, desktop computer, in-vehicle terminal, intelligent voice interaction device, smart home appliance, mobile device, aircraft, etc. Mobile devices can be various types of terminals such as smartphones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, etc. This application embodiment does not specifically limit this.

[0056] Server 20 can be a single server, a server cluster or distributed system consisting of several servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0057] Terminal 10 and server 20 establish a communication connection via a wired or wireless network. Optionally, the aforementioned wireless or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless network, private network or virtual private network, and any combination thereof.

[0058] For ease of understanding and explanation, the following will use... Figures 3 to 19 This application provides a detailed description of the traffic sign recognition method, apparatus, equipment, and medium provided in the embodiments of this application.

[0059] Figure 3 The diagram shown is a flowchart of a traffic sign recognition method according to an embodiment of this application. This method can be executed by a computer device, which can be the aforementioned... Figure 2 The system shown can be either server 20 or terminal 10, or the computer device can be a combination of terminal 10 and server 20. Figure 3 As shown, the method includes:

[0060] S101. Perform traffic sign detection on the traffic image to be detected to obtain the traffic sign region image corresponding to the traffic image to be detected.

[0061] The aforementioned traffic image to be detected can be an image obtained by capturing a panoramic view of traffic including traffic signs. This traffic image may include the traffic signs to be detected, and may also include background information. Traffic signs are the basic components of the image, used to identify warning situations in the geographical environment. Traffic signs may include illuminated warning signs, luminous warning signs, and reflective warning signs, such as traffic lights. Background information refers to image information in the traffic image to be detected other than the traffic signs to be detected, such as vehicles, roads, poles, buildings, sky, ground, trees, etc.

[0062] The traffic image to be detected can be an image taken in various different scenarios, such as an image containing traffic signs taken under different roads, different weather conditions, and different angles.

[0063] For example, the traffic signs mentioned above may include arrow types indicating straight ahead, left turn, right turn, left turn straight ahead, right turn straight ahead, U-turn, continuous downhill, etc., and may also include bus symbols indicating bus lanes, bicycle symbols indicating non-motorized vehicle lanes, car symbols indicating motorized vehicle lanes, and stair symbols indicating pedestrian overpasses and underpasses.

[0064] In this embodiment of the application, the traffic image to be detected can be obtained by calling an image acquisition device to acquire an image of a scene containing traffic signs, or by acquiring it through the cloud, a database or blockchain, or by importing it through an external device.

[0065] In one possible implementation, the aforementioned image acquisition device can be a camera or a camera, or it can be a radar device such as a lidar or millimeter-wave radar. The image acquisition device is an image acquisition device located in the vehicle, such as a dashcam or a camera mounted on the windshield of the vehicle with its lens facing the direction of travel.

[0066] The camera can be a monocular camera, a binocular camera, a depth camera, a 3D camera, etc. Optionally, during image acquisition via the camera, the camera can be controlled to activate its video recording mode, scan traffic signs in its field of view in real time, and capture images at a specified frame rate to obtain road video, which is then processed to generate the traffic image to be detected. Alternatively, during image acquisition via radar equipment, the radar equipment can transmit detection signals to traffic signs in real time, then receive the echo signals reflected back from the traffic signs. Based on the difference between the detection signal and the echo signal, the characteristic data of the traffic signs is determined, and based on this characteristic data, the traffic image to be detected is determined.

[0067] It should be noted that the traffic images to be detected can be in the format of image sequences, 3D point cloud images, or video images.

[0068] In one embodiment, after acquiring the traffic image to be detected, the computer device can perform traffic sign detection on the traffic image using preset feature extraction rules to obtain a traffic body region image corresponding to the traffic image to be detected. It should be noted that the aforementioned traffic body region image may include a traffic body target region, which refers to an image containing only traffic bodies. This target region can be a rectangular region, a circular region, a triangular region, etc.

[0069] Optionally, the aforementioned feature extraction rules refer to a pre-set feature extraction strategy for the image to be recognized, based on the actual application scenario. This can be a trained traffic body region prediction model or a general feature extraction algorithm. As one possible approach, the traffic body region prediction model can be used to extract features from the image to be recognized, obtaining the target region of the traffic sign within the traffic image to be detected. The traffic body region prediction model is trained on sample data to learn a network structure model capable of extracting traffic signs. The model takes the traffic image to be detected as input and outputs the target region of the traffic body within that image; that is, it outputs a traffic body region image and possesses the ability to recognize the traffic image to be detected. It is a neural network model capable of predicting traffic body region images. The traffic body region prediction model can include a multi-layered network structure. Different layers process the input data differently and transmit their outputs to the next layer until the final layer processes the data to obtain the target region of the traffic body within the traffic image to be detected.

[0070] It should be noted that the above-described implementations of detecting traffic signs in the traffic image to obtain the corresponding traffic area image are merely examples, and the embodiments of this application do not limit this.

[0071] In one embodiment, considering the limited computing resources provided by the computer device, in order to reduce the amount of computation in the image processing process, after acquiring the traffic image to be detected, the computer device can compress the directly acquired traffic image to be detected to obtain the processed traffic image to be detected.

[0072] Optionally, either lossy compression or lossless compression can be used to compress the high-resolution traffic image to be detected. The compression ratio can be improved by ignoring minor information that is not sensitive to human vision, so as to obtain the processed traffic image to be detected.

[0073] As one feasible approach, a resolution threshold can be pre-set based on the actual processing requirements of computing resources. After acquiring the traffic image to be detected, its resolution can be determined. If the resolution of the traffic image to be detected is less than the resolution threshold, no compression is required, and subsequent processing can proceed directly. If the resolution of the traffic image to be detected is not less than the resolution threshold, then compression is necessary to obtain a processed traffic image before proceeding with subsequent processing.

[0074] In this embodiment, by compressing the traffic image to be detected, the amount of data represented by the traffic image can be reduced, requiring less computing resources for image recognition processing, thereby saving computing resources.

[0075] In this embodiment, by performing traffic sign detection on the traffic image to be detected, the target area of ​​the traffic body in the traffic image to be detected can be accurately obtained, that is, the traffic body area image can be obtained. This allows for the acquisition of more fine-grained image features of the traffic sign area image for the correct traffic body area image, thereby making the determined traffic sign recognition result more accurate.

[0076] S102. Determine N anchor frames in the traffic body region image, perform traffic sign detection on the region corresponding to each anchor frame, and obtain the traffic sign region image corresponding to the traffic sign body region image; the size and position of each of the N anchor frames are different, and N is an integer greater than or equal to 1.

[0077] It should be noted that the above anchor boxes are used to select a preset region in the model input image, such as a traffic body region. The anchor box refers to multiple prior boxes defined by a preset algorithm with the anchor point as the center. The shape of the prior box can be, for example, a rectangle, a triangle, a rhombus, a circle, etc.

[0078] The aforementioned N anchor boxes are obtained through training based on historical traffic body images and the ground truth classification results of those images. The historical traffic body images contain labeled traffic body regions. When the prior box is rectangular, it can be multiple prior boxes of different sizes and aspect ratios generated with each pixel as the center. These sizes and aspect ratios are obtained through training based on historical traffic body images and the ground truth classification results of those images. The aforementioned historical traffic body images contain labeled traffic body regions. When the prior box is circular, it can be multiple prior boxes of different radii generated with each pixel as the center. These radii are obtained through training based on historical traffic body images and the ground truth classification results of those images.

[0079] As an optional implementation, assuming the width and height of the historical traffic image are w and h respectively, anchor frames of different shapes are generated centered on each pixel of the image. Let the size be s∈(0,1] and the aspect ratio be r>0. Then, when the center position is determined, the corresponding anchor frame can be determined. For example, a set of S1, S2, ..., S... can be defined. n And a set of aspect ratios r1, r2, ..., r mIf we use all combinations of size and aspect ratio centered at each pixel, we can obtain whnm anchor boxes. Although these anchor boxes may cover all the real bounding boxes, the computational complexity can easily become too high. Therefore, we are usually only interested in combinations of size and aspect ratio containing S1 and r1, i.e.: (S1, r1), (S1, r2), (S1, r... m ), (S2, r1), (S3, r1),..., (S n If the number of anchor boxes centered on the same pixel is n+m-1, then for the entire historical traffic volume image, a total of wh(n+m-1) anchor boxes are generated.

[0080] As another possible approach, historical traffic volume images can be obtained, and then the aspect ratios of the target boxes in the historical traffic volume images can be counted. The k-means clustering method can be used to generate k aspect ratio values. The aspect ratios of the anchor boxes correspond to the k values ​​generated by the clustering, thus obtaining k types of anchor boxes with different shapes.

[0081] Specifically, after obtaining the traffic body region image corresponding to the traffic image to be detected, traffic sign detection can be performed on the traffic body region image. For example, traffic sign detection can be processed using a feature detection strategy to obtain the traffic sign region image corresponding to the traffic sign body region image. Optionally, this feature detection strategy refers to a strategy pre-set according to the actual application scenario for feature extraction and detection. It can be a detection network that has been trained, or a general feature extraction algorithm, etc. As one possible approach, traffic sign detection can be performed on the traffic body region image using a detection network to obtain the image features of the traffic sign region image corresponding to the traffic sign body region image. This detection network is trained on sample data to learn a network structure model with feature extraction capabilities. The input of the detection network is each sub-region within multiple sub-regions, and the output is the image features of each sub-region. It also has the ability to perform image recognition on each sub-region and is a neural network model capable of predicting the image features of each sub-region. This detection network can include a multi-layer network structure. Different layers of the network structure process the input data differently and transmit their output results to the next network layer until the last network layer processes the data to obtain the image features of each sub-region.

[0082] As another possible approach, traffic signs can be detected in the traffic body region image by using a feature extraction algorithm to obtain the image features of the traffic sign region image corresponding to the traffic sign body region. This feature extraction algorithm can be, for example, the Scale-Invariant Feature Transform (SIFT) algorithm, the Speeded Up Robust Features (SURF) algorithm, or the Oriented Fast and Rotated BRIEF (ORB) feature detection algorithm, etc.

[0083] It is understandable that extracting features from different traffic sign regions will result in different image features for the traffic sign region images.

[0084] As another feasible approach, a pre-established template image database can be consulted. The image features of the traffic body area image are compared with those in the template image database to identify the portions of the traffic body area image that match the template image features in the database. These matching portions are then designated as the target areas for the traffic signs within the traffic body area image. The template image database can be flexibly configured based on the image feature information of traffic signs in the actual application scenario. It is constructed by summarizing and organizing traffic signs with different traffic body types, shapes, and structures.

[0085] It should be noted that the above-described methods for detecting traffic signs in traffic body area images to obtain traffic sign area images corresponding to traffic sign body area images are merely examples, and the embodiments of this application do not limit them.

[0086] In this embodiment, by performing traffic sign detection on the traffic body area image, the image features of each traffic sign area can be extracted in a more granular manner, thereby further improving the accuracy of traffic sign recognition results.

[0087] S103. Perform sign type detection on the traffic sign area image to obtain the type information of traffic signs in the traffic image to be detected, and obtain the traffic sign recognition result of the traffic image to be detected based on the type information.

[0088] It should be noted that the above-mentioned sign type detection is used to detect the type information of traffic signs in the traffic sign area image. This type information includes the shape, color, and status information of the traffic sign. Optionally, the shape information may be, for example, straight ahead, left turn, right turn, or U-turn; the color information may be, for example, red, yellow, and green; and the status information may be, for example, a stationary state or a moving state.

[0089] Specifically, after obtaining the traffic sign area image, the image can be input into a trained classification network to determine the traffic sign recognition result of the traffic image to be detected. Alternatively, a classification algorithm can be used to determine the traffic sign recognition result.

[0090] It should be noted that the above classification network is a model structure with traffic sign recognition capability learned through samples. The input of the classification network is the traffic sign area image, and the output is the traffic sign recognition result of the traffic image to be detected. It is a neural network model that can predict the traffic sign recognition result.

[0091] As one possible approach, the classification network can include fully connected layers and activation functions. After acquiring the traffic sign area image, the fully connected layers can be used to process the traffic sign area image to obtain fully connected vector features. The activation functions can then be used to process the fully connected vector features to obtain the traffic sign recognition result. This recognition result can include multiple traffic sign types, or multiple type attributes under each traffic sign type.

[0092] As another possible approach, feature information from traffic sign area images can be extracted using prior knowledge in the relevant domain. This feature information is then clustered using a clustering algorithm to obtain clustering results. Finally, prior knowledge of the sign features is used to determine the recognition result for each cluster, thus obtaining the traffic sign recognition result. The clustering algorithm can be a clustering function, such as Mean, Pool, or LSTM.

[0093] As another feasible approach, a pre-established database of known traffic sign types can be queried. The feature information of unknown types extracted from the traffic sign area image is compared with the features in the database of known sign types. Sign types with the same features are identified as the traffic sign recognition results for the traffic image to be detected. This feature database can be constructed by summarizing and classifying sign data with different traffic sign types, shapes, and structures. The aforementioned traffic sign recognition results are used to identify traffic signs in the traffic image to be detected, enabling the rapid acquisition of information and characteristics of the traffic signs.

[0094] For example, the traffic sign recognition result of the traffic image to be detected may include the traffic sign type of the traffic image, or it may include multiple traffic sign attributes under the traffic sign type. For instance, the traffic sign type may be a motor vehicle traffic light, a non-motor vehicle traffic light, a pedestrian crossing traffic light, etc. Among these, the traffic sign attributes corresponding to motor vehicle traffic lights may be, for example, a green light for going straight, a red light for turning left, a green light for turning right, etc. Different traffic sign attributes correspond to different functions of the traffic signs. For example, a green light for going straight indicates that vehicles can go straight; a red light for turning left indicates that vehicles are prohibited from turning left; and a green light for turning right indicates that vehicles can turn right.

[0095] Please see Figure 4 As shown, when the traffic image to be detected 3-1 is obtained, traffic sign body detection can be performed on the traffic image to be detected 3-1 to obtain the traffic body region image 3-2 corresponding to the traffic image to be detected. Then, traffic sign detection can be performed on the traffic body region image 3-2 to obtain the traffic sign region image 3-3 corresponding to the traffic sign body region image 3-2. Then, sign type detection can be performed on the traffic sign region image 3-3 to obtain the type information 3-4 of the traffic signs in the traffic image to be detected. Based on the type information 3-4, the traffic sign recognition result 3-5 of the traffic image to be detected is obtained.

[0096] The traffic sign recognition method provided in this application, compared with the prior art, has several advantages. First, after obtaining the guidance information of the traffic body area image, it determines N anchor boxes in the traffic body area image, facilitating more precise traffic sign detection for each anchor box's corresponding area. This allows for finer-grained extraction of image features for each traffic sign area in the traffic body area image, enabling more detailed feature recognition and effectively improving the accuracy of traffic sign recognition. Second, by performing sign type detection on the traffic sign area image, it combines more refined and comprehensive features to determine the traffic sign recognition result of the traffic image to be detected, significantly improving the accuracy of traffic sign recognition results in the traffic image to be detected. It can also be applied to autonomous driving systems to accurately predict traffic signs in the traffic image to be detected, greatly improving the quality and efficiency of traffic sign recognition and providing strong support for accurate navigation and road condition analysis in autonomous driving.

[0097] In another embodiment of this application, traffic sign detection can be performed on the traffic body area image to obtain the traffic sign area image corresponding to the traffic sign body area image. Figure 5 This provides a specific implementation method for obtaining the traffic sign area image corresponding to the traffic sign body area image. Please refer to [link / reference]. Figure 5 As shown, it specifically includes:

[0098] S201. Input the traffic body region image into the detection network, extract features from the region corresponding to each anchor frame, and obtain the position offset and size offset of each anchor frame in the traffic body region image.

[0099] It should be noted that the aforementioned detection network is a neural network model that takes a traffic area image as input and outputs the recognition results of the positional and dimensional offsets of the anchor frames. It has the capability to detect traffic areas within the image and predict the positional and dimensional offsets of each anchor frame. This detection network is responsible for establishing the relationship between the traffic area image and the positional and dimensional offsets of the anchor frames, and its model parameters are already optimized.

[0100] The detection network may include, but is not limited to, convolutional layers, normalization layers, and activation functions. These layers may consist of a single layer or multiple layers. The convolutional layers are used to extract edge and texture features from the traffic area image. The normalization layers normalize the image features obtained from the convolutional layers; for example, the image features can be normalized by subtracting the mean and dividing by the variance to obtain a normal distribution with a mean of zero and a variance of one, preventing gradient explosion and gradient vanishing. The activation function can be a Sigmoid function, a Tanh function, or a ReLU function. By processing the normalized feature map through the activation function, the result can be mapped to the range of 0 to 1.

[0101] In one possible implementation, as shown in Figure 6, the detection network can include a basic convolutional module and a separable convolutional module. The traffic volume region image 6-1 can be input into the detection network 6-2. The basic convolutional module in the detection network 6-2 extracts features from the anchor boxes to obtain a feature map. The separable convolutional module then performs convolution processing to obtain output parameters 6-3. These output parameters include the positional and size offsets corresponding to each extracted anchor box class, as well as the confidence score and category result determined by an activation function for the preset box. The confidence score characterizes the probability of a traffic sign existing within the preset box, and the category result indicates whether a traffic sign exists within the preset box.

[0102] It is understood that the aforementioned positional offset may include the horizontal and vertical coordinate offsets of the center point of the preset frame relative to the anchor frame, and the aforementioned dimensional offsets may include the width and height offsets of the preset frame relative to the anchor frame.

[0103] S202. Determine the preset frame corresponding to the anchor frame based on the anchor frame's position, anchor frame size, anchor frame's corresponding position offset, and size offset.

[0104] Specifically, after obtaining the positional offset and dimensional offset of the anchor frame, the anchor frame can be corrected based on its position, size, and corresponding positional and dimensional offsets to obtain a preset frame. A preset correction algorithm can be used for this process. For example, when determining the position of the preset frame, the anchor frame's position can be added to its corresponding positional offset; similarly, when determining the size of the preset frame, the size can be added to its corresponding dimensional offset, thus obtaining the position and size of the preset frame.

[0105] For example, assuming the position offset (center point horizontal and vertical coordinate offset) of the anchor frame is determined to be (Δx, Δy) and the size offset (width offset and height offset) is determined to be Δw and Δh respectively, and the position (center point horizontal and vertical coordinates) of the anchor frame is determined to be (x1, y1), and the size (width and height) of the anchor frame is determined to be w1 and h1 respectively, then the position coordinates of the preset frame corresponding to the anchor frame can be determined to be (x1+Δx, y1+Δy), the width of the preset frame is Δw+w1, and the height of the preset frame is Δh+h1, thus determining the preset frame corresponding to the anchor frame.

[0106] Furthermore, for each feature point on the traffic area image, the aforementioned correction algorithm can be applied to each type of anchor box to obtain a preset box corresponding to each type of anchor box. Therefore, N preset boxes corresponding to N anchor boxes can be obtained. These N preset boxes may include boxes containing traffic signs or boxes without traffic signs.

[0107] S203. Filter out the invalid preset frames in the preset frames corresponding to the N types of anchor frames, and obtain the traffic sign area image based on the area corresponding to the remaining anchor frames.

[0108] It should be noted that the remaining anchor boxes mentioned above refer to the areas selected by the anchor boxes. The invalid preset boxes mentioned above refer to preset boxes that deviate too much from the true predicted boxes. The true preset boxes refer to preset boxes that only contain the traffic sign area. Invalid preset boxes can be, for example, preset boxes that do not completely contain the traffic sign area, or preset boxes that contain not only the traffic sign area but also the background area. Here, the background refers to the rest of the image in the traffic volume area image besides the traffic signs, such as trees, sky, roads, poles, vehicles, etc.

[0109] Understandably, during traffic sign detection, a large number of bounding boxes are generated at the same target location. These bounding boxes may overlap. Therefore, a non-maximum suppression (NMS) algorithm is needed to eliminate redundant and invalid bounding boxes and determine the optimal target bounding box. This means determining the traffic sign region image based on the area selected by the remaining bounding boxes. For example, the DIoU-NMS algorithm can be used for this purpose.

[0110] As an optional implementation, the above N preset boxes can be categorized and summarized to obtain a preset box list and its corresponding confidence score list. The non-maximum suppression algorithm can then proceed as follows: obtain the preset box list and its corresponding confidence score list, and set a threshold to delete preset boxes with significant overlap. First, sort the preset boxes according to their confidence scores, select the preset box with the highest confidence score and add it to the final output list, then delete it from the preset box list. Calculate the area of ​​all preset boxes, and calculate the overlap ratio (area ratio of the overlapping region) between the preset box with the highest confidence score and other preset boxes based on the area of ​​the preset boxes. Delete preset boxes with an IoU greater than the threshold, and repeat the above process until the preset box list is empty.

[0111] For example: There are six rectangles in the preset list of boxes. Assume that the confidence scores of the six rectangles are in ascending order and their corresponding confidence scores are A, B, C, D, E, and F. In the non-maximum suppression algorithm, we can start with the confidence F of the highest confidence rectangle. We then determine whether the overlap (IoU) between the rectangles with confidence levels A to E and the rectangle with confidence level F is greater than a set threshold. If the overlap between the rectangles with confidence levels B and D and the rectangle with confidence level F exceeds the threshold, we remove the rectangles with confidence levels B and D, and mark and retain the first rectangle with confidence level F. Then, from the remaining rectangles with confidence levels A, C, and E, we select the rectangle with the highest confidence level E, and then determine the overlap between the rectangle with confidence levels E and the rectangles with confidence levels A and C. If the overlap is greater than the set threshold, we remove the rectangles with confidence levels A and C, and mark and retain the second rectangle with confidence level E. By repeating the above process, we can determine the retained rectangles and obtain the corresponding traffic area image.

[0112] In this embodiment, by inputting the traffic body area image into the detection network, the position offset and size offset corresponding to each type of anchor frame can be accurately determined. Thus, based on the position of the anchor frame, the size of the anchor frame, and the position offset and size offset corresponding to the anchor frame, the preset frame corresponding to the anchor frame can be accurately determined, thereby improving the accuracy of determining the traffic sign area image.

[0113] In another embodiment of this application, feature extraction can be performed on the traffic area image to obtain the positional and dimensional offsets corresponding to each type of anchor frame. Please refer to [link to relevant documentation]. Figure 7 As shown, the first feature can be obtained by performing basic convolution processing on the traffic body region image 7-1 through the basic convolution module 7-2. Figure 7-3 The basic convolutional processing includes convolution, normalization, and activation function processing in sequence; and the first feature... Figure 7-3 The second feature is obtained by performing separable convolution processing through separable convolution module 7-4. Figure 7-5 Then, for the second feature Figure 7-5 Basic convolution processing is performed through the basic convolution module 7-6 to obtain output parameters 7-7, which may include position offset and size offset.

[0114] Specifically, after acquiring the traffic body region image, the target region of the traffic body region image can be determined based on the anchor box. This target region is the area selected by the anchor box in the traffic body region image. Then, the target region is processed by a basic convolutional module sequentially through convolution, normalization, and activation function to obtain the first feature map. The basic convolutional module can be a ConvBnReLU module, which contains a Conv convolutional layer, a Bn layer, and a ReLU layer connected in sequence. The Conv convolutional layer is used to extract image features such as edges and textures from the anchor box in the traffic body region image, obtaining the image features of the traffic body region image. The Bn layer is used to normalize the image features extracted by the Conv convolutional layer. Its input is the output feature of the Conv convolutional layer, filtering out noise features in the image features to obtain filtered image features. Then, the filtered image features are linearly mapped through an activation function to enhance the generalization ability of the feature extraction model, obtaining the first feature map. The first feature map includes multiple feature points.

[0115] After obtaining the first feature map, it can be processed by a separable convolutional module, which includes one or more DW-PW convolutional layers. The first feature map can be processed sequentially by DW convolution and PW convolution, where DW convolution is used for filtering and PW convolution is used for channel transformation. The DW-PW convolutional layer can include a depthwise layer, a BN layer, and a ReLU layer, performing convolution processing, normalization, and activation function processing in sequence. This results in the normalized weighted features being weighted onto the features of each channel to obtain the second feature map. Then, basic convolution processing is performed on the second feature map, sequentially extracting features from the second feature map through a Conv convolutional layer. The BN layer is then used to normalize the features extracted by the Conv convolutional layer, with its input being the output features of the Conv convolutional layer. The normalized features are then processed by an activation function to obtain output parameters. These output parameters can include the position offset and size offset of the predicted bounding box relative to the anchor box, the confidence score of the predicted bounding box, and the category result of the predicted bounding box.

[0116] Optionally, in separable convolution processing, the number of DW-PW convolutional layers can be set to multiple or one. The more DW-PW convolutional layers there are, the more information of different granularities can be extracted, resulting in more comprehensive information of the extracted feature map, and thus more accurate position and size offsets.

[0117] It's important to note that the Depthwise Convolution (DW) described above differs from regular convolution operations. In Depthwise Convolution, one kernel is responsible for one channel, and each channel is convolved by only one kernel. In contrast, in regular convolution, each kernel operates on every channel of the input image simultaneously.

[0118] For example, for a 5×5 pixel, three-channel color input image (shape 5×5×3), Depthwise Convolution first performs a convolution operation. Unlike regular convolution, DW is performed entirely in a two-dimensional plane. The number of convolution kernels is the same as the number of channels in the previous layer (channels and convolution kernels correspond one-to-one), so a three-channel image generates three feature maps after the operation.

[0119] The number of feature maps after Depthwise Convolution is the same as the number of channels in the input layer, making it impossible to expand the number of feature maps. Furthermore, this operation performs convolution independently on each channel of the input layer, failing to effectively utilize feature information from different channels at the same spatial location. Therefore, Pointwise Convolution (PW) is needed to combine these feature maps to generate new feature maps.

[0120] Pointwise Convolution operates very similarly to regular convolution, with a kernel size of 1×1×M, where M is the number of channels in the previous layer. Therefore, this convolution operation weights and combines the previous map along the depth direction to generate a new feature map. The number of output feature maps corresponds to the number of convolution kernels. In other words, the Depthwise layer only changes the size of the feature map, not the number of channels. Conversely, the Pointwise layer only changes the number of channels, not the size.

[0121] In this embodiment, the target region of the traffic body area image is determined based on the anchor frame, and basic convolution and separable convolution processing are performed on the target region to extract more fine-grained information from the image. Furthermore, the DW and PW convolution processing in the separable convolution can reduce the number of parameters and improve the detection speed.

[0122] In another embodiment of this application, before performing traffic sign detection on the traffic body area image, the size of the traffic body area image can be adjusted according to preset size parameters.

[0123] Specifically, the size and shape of the traffic body region images obtained above may be different, and the traffic body region images need to be resized to make them the size and shape that meet the input requirements of the detection network.

[0124] This process involves acquiring the location and size information of the traffic body area image, then scaling and expanding the image based on the location and size information of the traffic body area image according to preset size parameters, and mapping the coordinate position of the traffic body area.

[0125] It should be noted that the aforementioned preset size parameters can be customized according to the actual input requirements of the detection network, and may include preset size information or position information. The position and size information of the traffic body area image can be obtained. When the position and size information of the traffic body area image does not conform to the preset parameters, image scaling (resize) and padding are required for the traffic body area image.

[0126] Optionally, the image scaling described above can be achieved using the resize function to change the image size. For example, it can be implemented using the following interpolation methods: nearest neighbor interpolation, cubic spline interpolation, linear interpolation, and region interpolation.

[0127] As one feasible approach, when expanding the image of a traffic vehicle area, pre-set software can be used. Users can customize relevant parameters according to their needs, such as user-defined options like "expansion" and "size." The pre-set software is then run to expand the traffic sign area of ​​the traffic vehicle image according to these parameters, resulting in a processed image that meets the input requirements of the detection network. The pre-set software can be image processing software.

[0128] When expanding the size of a traffic sign area, the expansion can be performed with the traffic sign area as the center, according to a preset ratio. This preset ratio is customized according to actual needs; for example, the preset ratio of the expansion of the traffic sign area to the processed image can be any value such as 1:2, 1:3, 1:4, 2:3, etc.

[0129] For example, when the size of the traffic sign area in the traffic body area image is determined to be W×H, assuming that the target area is expanded according to the preset ratio of 1:2 to obtain the processed image, the size of the processed image is 2W×2H, and the processed image includes the traffic sign area.

[0130] For example, after acquiring the traffic body area image, it can be determined whether the image needs adjustment based on preset model input rules. If no adjustment is needed, feature extraction can be performed directly on the traffic body area image to obtain the positional and dimensional offsets corresponding to each type of anchor frame. Please refer to [link to relevant documentation]. Figure 8As shown, when adjustments are needed, preset size parameters can be selected according to actual requirements. For example, a preset side length s can be selected. Then, the traffic sign area image is scaled and resized according to a preset ratio. After the image is scaled and resized, the long side is s. For the short side of the traffic sign area image, it can be expanded according to a preset ratio with the traffic sign area as the center. For example, the image can be expanded symmetrically on both sides with padding so that the size of the processed traffic sign area image is s*s. Then, according to the mapping relationship between the traffic sign area image and the processed traffic sign area image, the coordinates of the traffic sign area are mapped to the processed traffic sign area image of size s*s. Thus, feature extraction can be performed through the processed traffic sign area image to obtain the position offset and size offset of each type of anchor frame.

[0131] In another embodiment of this application, a specific implementation of the training process for training the detection network is also provided. Please refer to [link to relevant documentation]. Figure 9 As shown, it specifically includes:

[0132] S301. Obtain M anchor boxes from the historical traffic volume image; each anchor box has a different size and position, M is an integer greater than or equal to 1, and the historical traffic volume image contains the labeled traffic volume area.

[0133] Specifically, the aforementioned historical traffic image may be multiple or a single image. Each historical traffic image may include at least one traffic sign, such as a traffic light sign for motor vehicles, a traffic light sign for non-motor vehicles, or a traffic light sign for pedestrians.

[0134] After obtaining historical traffic body images, the size and shape of these images vary and do not meet the input requirements for training the first initial network. Therefore, they need to be processed. The size of the historical traffic body images can be adjusted according to preset parameters. For example, a fixed side length s can be selected, and the traffic body image can be scaled according to a preset ratio so that the long side of the scaled image is s. For the short side, the image can be symmetrically expanded on both sides of the traffic body region, so that the size of the processed image is s*s. Based on the mapping relationship between the processed image and the historical traffic body images, the coordinates of the traffic sign region are mapped to the processed image of size s*s, thereby obtaining a historical traffic body image that meets the input requirements of the first initial network.

[0135] Then, the anchor frames of the historical traffic images are determined. The width and height of the historical traffic images can be used as detection features. Based on these features, the k-means algorithm is used to cluster the traffic images, resulting in M ​​clusters. The anchor frame size is determined according to the length and width corresponding to the position coordinates of the center point of each cluster. The center of each of the M clusters is then used as the width and height of the corresponding anchor frame. The size and position of the anchor frames differ between different clusters.

[0136] S302. Input the historical traffic body image into the first initial network, extract features from the historical traffic body image, and obtain the position offset and size offset of each type of anchor frame in the M anchor frames.

[0137] It should be noted that the aforementioned first initial network is a neural network model that takes a historical traffic body image as input and outputs the positional and size offsets of each type of anchor box. It also possesses the ability to extract features from the historical traffic body image and predict the positional and size offsets of each type of anchor box within the historical traffic body image. This first initial network can be the initial model during iterative training, meaning its model parameters are in their initial state, or it can be a model adjusted in the previous iteration, meaning its model parameters are in an intermediate state. The historical traffic body image can be input into the first initial network to obtain the output results. These output results can include the positional and size offsets of the predicted box relative to each type of anchor box, the confidence score of the predicted box, and the category result of the predicted box. The category result characterizes whether the predicted box contains a traffic sign, and the confidence score characterizes the probability that the image within the predicted box is a traffic sign.

[0138] After inputting the historical traffic image into the first initial network, the historical traffic image can be processed by convolution, normalization and activation function in sequence through the basic convolution module to obtain the corresponding first sample feature map. Then, the sample feature map is processed by convolution through the separable convolution module to obtain the second sample feature map. The second sample feature map is then processed by convolution, normalization and activation function in sequence through the basic convolution module to obtain the output result of the first initial network.

[0139] The basic convolutional module can include a Conv convolutional layer, a Bn layer, and a ReLU layer connected sequentially. The Conv convolutional layer extracts image features such as the edges and textures of each anchor box in the historical traffic image, resulting in image features of the historical traffic image. Then, the Bn layer normalizes the image features extracted by the convolutional layer according to a normal distribution to filter out noise features, resulting in filtered image features. An activation layer then performs a non-linear mapping on the filtered image features to enhance the generalization ability of the feature extraction model, yielding a first sample feature map. This first sample feature map includes multiple feature points. The aforementioned separable convolutional module can include a DW convolutional layer and a PW convolutional layer.

[0140] S303. For each of the M anchor frames, determine the prediction frame corresponding to the anchor frame based on the anchor frame's position, size, position offset, and size offset.

[0141] S304. Filter out the invalid preset boxes in the prediction boxes corresponding to the M anchor boxes to obtain the predicted traffic body region of the first initial network.

[0142] Specifically, for each of the M anchor frames, after obtaining the positional and dimensional offsets corresponding to the anchor frame, the anchor frame can be corrected based on its position, size, and corresponding positional and dimensional offsets to obtain a preset frame, thus obtaining M predicted frames. Then, a non-maximum suppression algorithm is used to filter out invalid predicted frames from the M predicted frames, thereby obtaining the predicted traffic body region of the first initial network.

[0143] S305. Based on the loss function between the predicted traffic body region and the labeled traffic body region, the first initial network is iteratively trained using an iterative algorithm to minimize the loss function, thereby obtaining the detection network.

[0144] After obtaining the output results, a loss function can be constructed based on the output results and the annotation results of the traffic body region. The first initial network to be constructed is optimized according to minimizing the loss function to obtain the detection network. The parameters in the first initial network to be constructed are updated according to the difference between the output results and the annotation results to achieve the purpose of training the first initial network. The annotation results mentioned above can be the recognition results obtained by manually annotating historical traffic body images.

[0145] Optionally, updating the parameters in the first initial network to be constructed may involve updating matrix parameters such as the weight matrix and bias matrix in the first initial network to be constructed. The weight matrix and bias matrix include, but are not limited to, the matrix parameters in the convolutional layers, feedforward layers, and fully connected layers of the first initial network to be verified.

[0146] When updating the parameters of the initial network to be validated using the loss function, if the initial network has not converged according to the loss function, the parameters in the model are adjusted to make the initial network converge, thus obtaining the detection network. Convergence of the initial network to be validated can be defined as the difference between the output and labeled results being less than a preset threshold, or the rate of change of the difference between the output and labeled results approaching a certain low value. When the calculated loss function is small, or the difference between it and the loss function output from the previous iteration approaches 0, the initial network to be validated is considered converged, and the detection network is obtained.

[0147] In another embodiment of this application, a specific implementation of the loss function of the first prediction model is also provided. In one possible implementation, the loss function of the first initial model includes a first component, a second component, a third component, and a fourth component. The first component is used to characterize the positional offset loss between the predicted box and the anchor box; the second component is used to characterize the dimensional offset loss between the predicted box and the anchor box; the third component is used to characterize the loss between the predicted confidence and the true confidence corresponding to the predicted box; and the fourth component is used to characterize the loss between the predicted class result and the true class result corresponding to the predicted box.

[0148] The loss function constructed above can be the sum of a first component, a second component, a third component, and a fourth component. By setting the first and second components in the loss function, the difference between the positional offset between the predicted bounding box and the anchor box corresponding to the historical traffic body image input by the first initial model, as well as the difference between the size offset between the predicted bounding box and the anchor box corresponding to the historical traffic body image input by the first initial model, can be reduced. This allows for a comprehensive determination of the positional and size offsets of the predicted bounding box relative to the anchor box. Furthermore, by setting the third and fourth components, guidance information can be provided for determining the final predicted traffic body region, resulting in a more optimized detection network.

[0149] In this embodiment, when constructing the loss function to obtain the detection network, the differences between the true category results and the predicted category results of the first initial model, the positional offset differences between the predicted bounding box and the anchor box corresponding to the historical traffic body image input to the first initial model, and the size offset differences between the predicted bounding box and the anchor box corresponding to the historical traffic body image input to the first initial model are considered. Based on this loss function, the first initial model can be trained more accurately and comprehensively through iterative training of the model parameters in the first initial model, resulting in a better detection network and thus higher accuracy of the determined traffic sign area image.

[0150] In another embodiment of this application, during model training, reasonable weight coefficients can be assigned to the first, second, third, and fourth components of the loss function to ensure that the model's prediction discrepancies closely match actual business needs and improve model performance. In one possible implementation, when determining the loss function, the weight coefficients of the first, second, third, and fourth components can be determined, and the loss function can be determined based on these weight coefficients. Specifically, the weight coefficient of the first component is related to the importance of the positional offset between the predicted bounding box and the anchor box; the weight coefficient of the second component is related to the importance of the dimensional offset between the predicted bounding box and the anchor box; the weight coefficient of the third component is related to the importance of the confidence level corresponding to the predicted bounding box; and the weight coefficient of the fourth component is related to the importance of the predicted category result corresponding to the predicted bounding box.

[0151] The weight coefficient of the first component is positively correlated with the importance of the positional offset between the predicted box and the anchor box; that is, the larger the weight coefficient of the first component, the higher the importance of the positional offset between the predicted box and the anchor box. Similarly, the weight coefficient of the second component is positively correlated with the importance of the dimensional offset between the predicted box and the anchor box; that is, the larger the weight coefficient of the second component, the higher the importance of the dimensional offset between the predicted box and the anchor box. The weight coefficient of the third component is positively correlated with the importance of the confidence score corresponding to the predicted box; that is, the larger the weight coefficient of the third component, the higher the importance of the confidence score corresponding to the predicted box. The weight coefficient of the fourth component is positively correlated with the importance of the predicted category result corresponding to the predicted box; that is, the larger the weight coefficient of the fourth component, the higher the importance of the predicted category result corresponding to the predicted box.

[0152] The first component, second component, third component, fourth component, and loss function satisfy the following formula:

[0153] Y = a1*y1 + a2*y2 + a3*y3 + a4*y4

[0154] Where Y is the loss function in the first initial model training process, y1 is the first component, a1 is the weight coefficient of the first component, y2 is the second component, a2 is the weight coefficient of the second component, y3 is the third component, a3 is the weight coefficient of the third component, and y4 is the fourth component, a4 is the weight coefficient of the fourth component.

[0155] In this embodiment, the weight coefficients of the first, second, third, and fourth components can be reasonably determined according to the business scenario of the traffic sign recognition method application. One possible implementation involves determining the positional offset, size offset, prediction confidence level, and importance ratio of the prediction category results between the prediction box and the anchor box based on business requirements, and then determining the weight coefficients of the first, second, third, and fourth components based on the importance ratio.

[0156] For example, the loss function described above can be expressed by the following formula:

[0157]

[0158] In the above formula, the first row of parameters represents the positional offset loss of the predicted bounding box relative to the anchor box, the second row of parameters represents the dimensional offset loss of the predicted bounding box relative to the anchor box, S represents the width and height of the output feature map, and B represents the number of anchor boxes at each position in the output feature map. The first line indicates whether a traffic sign exists at position ij in the output feature map; the value is 1 if it exists, and 0 otherwise. The third line represents the confidence loss; the fourth line represents the class loss. α, β, and γ represent the weight coefficients of the first, second, third, and fourth components, respectively. ij The x-coordinate of the true center point of the anchor box at position ij in the feature map is represented by y. ij W represents the ordinate of the true center point of the feature map at position ij. ij h represents the width of the anchor box corresponding to position ij in the feature map. ij C represents the height of the anchor frame at position ij. ij denoted by , p(k) represents the true confidence score of the anchor box at position ij, and p(k) represents the true class result. This represents the x-coordinate offset of the predicted bounding box relative to the center point of the anchor box. This represents the offset of the center point of the predicted bounding box relative to the anchor box. This indicates the width offset of the prediction box relative to the anchor box. This indicates the height offset of the preset frame relative to the anchor frame. This indicates the confidence level of the predicted bounding box. This indicates the predicted category result of the prediction box.

[0159] In this embodiment, a loss function is constructed by setting a first component, a second component, a third component, and a fourth component. By optimizing the four parts of the loss function, the model parameters in the first initial network can be accurately adjusted, thereby making the detection network more optimal and enabling more accurate recognition of traffic signs.

[0160] In another embodiment of this application, a method is also provided for performing sign type detection on a traffic sign area image to obtain the traffic sign recognition result of the traffic sign in the traffic image to be detected. Please refer to [link to relevant documentation]. Figure 10 As shown, it specifically includes:

[0161] S401. Cropping the traffic sign area image to obtain the traffic sign image.

[0162] It should be noted that the aforementioned traffic sign area image may include the traffic sign image itself, as well as the background area other than the traffic sign area. To improve the accuracy of traffic sign image recognition, after obtaining the traffic sign area image, it can be cropped to remove redundant background areas, thus obtaining the traffic sign image. This traffic sign image only contains the traffic sign area.

[0163] Specifically, in the process of cropping traffic sign area images, image recognition can be used to identify the background area within the traffic sign area image. Then, image processing software can be used to crop the background area, resulting in a traffic sign image containing only the traffic sign area. Alternatively, image recognition can be used to identify the traffic sign area within the image, followed by cropping the area using image processing software. This yields a traffic sign image containing only the traffic sign area. Furthermore, by adjusting the image to different sizes and scaling ratios, a traffic sign image suitable for input into a classification network can be obtained.

[0164] For example, please see Figure 11 As shown, generally speaking, since the traffic sign area is close to a square, the side length s1 of the target size can be defined according to actual needs. Then, the traffic sign area image is adjusted according to a preset scaling ratio so that the size of the traffic sign image is s1*s1.

[0165] S402. Input the traffic sign image into the classification network for sign type detection to obtain the pixel information and shape information of the traffic sign image.

[0166] S403. Based on pixel information and shape information, perform semantic prediction to obtain the predicted semantics of the traffic sign image, and determine the traffic sign recognition result of the traffic image to be detected based on the predicted semantics.

[0167] It should be noted that the above classification network was trained based on historical traffic sign area images and historical classification results of these images, which are labeled with traffic sign areas. This classification network is a neural network model that takes a traffic sign image as input and outputs the traffic sign recognition result. It has the ability to identify the type of traffic sign in the image and predict the recognition result.

[0168] This classification network is responsible for establishing the relationship between the features of a traffic sign image and the target traffic sign type. Its model parameters are already optimized. This classification network may include, but is not limited to, convolutional layers, fully connected layers, and activation functions. Convolutional and fully connected layers may consist of a single layer or multiple layers. Convolutional layers are used for feature extraction from the traffic sign image, while fully connected layers primarily classify the features processed by the convolutional layers. Specifically, the traffic sign image is processed through convolutional layers to obtain convolutional features. These convolutional features are then processed through fully connected layers to obtain fully connected vectors. Finally, these fully connected vectors are processed through activation functions to obtain the output of the detection network. This output includes the feature type of the traffic sign, or it may include multiple traffic sign attributes under the traffic sign type.

[0169] The activation function mentioned above can be a Sigmoid function, a Tanh function, or a ReLU function. By processing the fully connected vector through the activation function, its result can be mapped to the range of 0 to 1.

[0170] In one embodiment, see Figure 12 As shown, the classification network mentioned above may include a feature extraction network 12-2 and a target classification network 12-3. The traffic sign image 12-1 is processed sequentially through the feature extraction network 12-2 and the target classification network 12-3 to obtain the traffic sign recognition result 12-4. Specifically, the feature extraction network is used to detect the sign type of the traffic sign image, extracting pixel and shape information. The target classification network is used to perform semantic prediction based on the pixel and shape information, obtaining the predicted semantics of the traffic sign image, and determining the traffic sign recognition result of the traffic image to be detected based on the predicted semantics. The pixel information may include pixel information of different colors, such as red, green, and yellow. The shape information may include different shapes, such as straight, left turn, right turn, etc.

[0171] In one possible implementation, the aforementioned feature extraction network may include a basic convolutional module, a separable module, and a pooling module. The basic convolutional module may be a ConvBnReLU module, wherein the ConvBnReLU module contains sequentially connected Conv convolutional layers, Bn layers, and ReLU layers, which extract feature information of the traffic sign image. The aforementioned separable convolutional module may include DW-PW convolutional layers, which may include one or more, to perform finer-grained feature extraction based on feature information, and to perform feature dimensionality reduction through the pooling module to obtain pixel information and shape information.

[0172] It should be noted that the pooling module mentioned above may include a pooling layer, also known as undersampling or downsampling. Its function is to reduce feature dimensionality, compress the amount of data and parameters, reduce overfitting, and improve the model's fault tolerance. Specifically, it performs feature selection, reduces the number of features, avoids overfitting, and thus reduces the number of parameters.

[0173] Optionally, the pooling layer mentioned above can be either max pooling or mean pooling. Max pooling typically takes the maximum value of all neurons within a region. Mean pooling typically takes the average value of all neurons within a region.

[0174] In this embodiment, the maximum value of the local region can be extracted through the max pooling layer. It not only has a good sense of local small features, but also can eliminate non-maximum values ​​and reduce the computational complexity of upsampling. Therefore, as the network depth increases, the max pooling operation of the convolution kernel is alternated to further refine the image features.

[0175] The processing of the aforementioned target classification network can specifically include: performing calculations on pixel and shape information using a multi-classification function to output the traffic sign type. Alternatively, it can perform calculations on pixel and shape information using multivariate binary classification to output fused feature attributes. Optionally, the multi-classification function can be a softmax function, and the multivariate binary classification function can be multiple sigmoid functions, with one sigmoid function performing one binary classification prediction. The purpose of the multi-classification function is to incorporate nonlinear factors, as linear models lack sufficient expressive power, and it transforms continuous real-valued inputs into outputs between 0 and 1.

[0176] For example, pixel information and shape information are input into the target classification network. The prediction result of the target classification network can include any one of the traffic sign types such as "motor vehicle traffic light", "non-motor vehicle traffic light", and "pedestrian crossing traffic light". The prediction result can also include traffic sign attributes. For example, the sign attributes corresponding to the traffic sign type "motor vehicle traffic light" can be multiple of "green light for going straight", "red light for turning left", "red light for turning right", and "green light for turning right".

[0177] Taking three-class classification as an example, the output of a multi-class classification function is introduced. For instance, the multi-class classification function can predict traffic sign types as "motor vehicle traffic light," "non-motor vehicle traffic light," and "pedestrian crossing traffic light." The output of the target classification network can be represented by a vector, such as a 3*1 dimensional vector. Each element in the vector corresponds to a traffic sign type, and each element value represents the probability that the traffic sign belongs to the corresponding label type. Assuming the output vector of the multi-class classification function is [0.61, 0.31, 0.08], it means that the probability of the traffic sign being "motor vehicle traffic light" is 0.61, the probability of the traffic sign being "non-motor vehicle traffic light" is 0.31, and the probability of the traffic sign being "pedestrian crossing traffic light" is 0.08. The element value with the highest probability can be selected as the predicted result of the traffic sign, that is, "motor vehicle traffic light" is taken as the recognition result of the traffic sign.

[0178] Taking ternary binary classification as an example, we introduce the output of a multivariate binary classification function. For instance, the sign attributes predicted by the multivariate binary classification function can be represented as "green light for straight ahead," "red light for left turn," "green light for right turn," and "red light for right turn." Its output can be represented by a vector, such as a 4*1 dimensional vector. Each element in the vector corresponds to a sign attribute, and each element value represents the probability that the traffic sign corresponds to that attribute. Assuming the output vector of the ternary binary classification function is [0.51, 0.15, 0.22, 0.62], this means the probability of the traffic sign being "green light for straight ahead" is 0.51, the probability of the traffic sign being "red light for left turn" is 0.15, the probability of the traffic sign being "green light for right turn" is 0.22, and the probability of the traffic sign being "red light for right turn" is 0.62. Assuming a preset threshold of 0.5, the element values ​​with probabilities greater than the preset threshold are used as the predicted results for the traffic signs, i.e., "green light for straight ahead" and "red light for right turn" are used as the recognition results of the multivariate binary classification function.

[0179] In this embodiment, by inputting traffic sign images into a classification network for sign type detection, the pixel and shape information of the traffic sign images can be accurately determined, which greatly improves the accuracy of traffic sign recognition results and enables more precise recognition, thus achieving higher-precision traffic sign recognition.

[0180] In another embodiment of this application, a specific implementation of the training process for training the classification network is also provided. This training process includes: acquiring historical traffic sign region images; determining historical traffic sign images based on the historical traffic sign region images, wherein the historical traffic sign images are labeled with corresponding historical classification results; and then training the classification network based on the historical traffic sign images and the historical classification results.

[0181] Specifically, after acquiring the historical traffic sign area image, the historical traffic sign image can be cropped to obtain a cropped historical traffic sign area image. Then, the historical traffic sign area image can be resized and data-enhanced to obtain the historical traffic sign image. For example, the cropped historical traffic sign area image can be adjusted according to different sizes and scaling ratios, and data-enhanced processing can be performed to obtain the historical traffic sign image.

[0182] It's important to note that data augmentation is a technique that artificially expands a training dataset by generating more equivalent data from a limited dataset. In the field of computer vision, data augmentation algorithms can be broadly categorized into two types: the first is data augmentation based on image processing techniques, and the second is data augmentation algorithms based on deep learning.

[0183] The image processing-based data augmentation techniques mentioned above can include geometric transformations, color transformations, rotation / reflection transformations, noise injection, kernel filters, image blending, random erasure, scaling transformations, translation, flipping transformations, and cropping. The deep learning-based data augmentation algorithms mentioned above can include feature space augmentation, adversarial generation, GAN-based data augmentation, and neural style transfer.

[0184] Compared with the prior art, the cropped historical traffic area images in this application do not have shape restrictions on the background area of ​​traffic bodies. For rare types of traffic signs, it is easy to obtain them by data augmentation with other types of traffic signs of the same color, thereby obtaining a sufficient number of historical traffic sign images with balanced categories, which in turn makes the trained classification network more accurate.

[0185] In the process of training a classification network based on historical traffic sign images and historical classification results, historical traffic sign images can be input into a second initial network for sign type detection to obtain pixel and shape information of the historical traffic sign images. Semantic prediction is then performed based on the pixel and shape information of the historical traffic sign images to obtain the predicted semantics of the historical traffic sign images. The prediction result of the historical traffic sign images is determined based on the predicted semantics. Based on the prediction result of the historical traffic sign images and the historical classification results, a loss function is calculated. By minimizing the loss function, an iterative algorithm is used to iteratively adjust the parameters of the second initial network to obtain the classification network.

[0186] Specifically, after obtaining historical traffic sign images, the computer equipment can randomly divide them into a training set and a validation set according to a certain ratio. The training set is used to train a second initial network to obtain a trained classification network, while the validation set is used to validate the trained detection network to verify the performance of the classification network. Then, the historical traffic sign images from the training set are input into the second initial network for sign type detection. First, they are processed by an initial feature extraction network and an initial target classification network. The historical traffic sign images are processed by convolution, normalization, and activation functions through sequentially connected Conv convolutional layers, Bn layers, and ReLU layers in the initial feature extraction network to obtain output features. Then, the output features are processed by a DW-PW convolutional module and a pooling module for feature extraction and feature dimensionality reduction, thereby extracting the pixel information and shape information of the historical traffic sign images.

[0187] The pixel and shape information of historical traffic sign images are processed through an initial target classification network, followed by fully connected layers and activation functions to obtain sample fully connected vectors. These vectors are then further processed using activation functions to produce the corresponding output. The feature extraction network and target classification network to be constructed are trained using a training set to obtain the networks to be validated.

[0188] It should be noted that the aforementioned second initial model is a neural network model that takes historical traffic sign images as input and outputs predictions of those images. It possesses the ability to extract information from and classify historical traffic sign images, and can predict the predicted results of those images. This second initial model can be the initial model used during iterative training, meaning its parameters are in their initial state, or it can be a model adjusted in the previous iteration, meaning its parameters are in an intermediate state. Historical traffic sign images can be input into the second initial model to obtain the output. This output includes the traffic sign type of the historical traffic sign image, or it can include multiple traffic sign attributes under that type.

[0189] During the training of the classification network, the computer equipment uses the feature extraction network and target classification network to be validated in the validation set. The feature extraction network and target classification network are optimized by minimizing the loss function to obtain the feature extraction network and target classification network. Based on the difference between the classification network to be validated and the annotation results in the validation set, the parameters in the feature extraction network and target classification network to be constructed are updated to achieve the purpose of training the feature extraction network and target classification network. The annotation results mentioned above can be traffic sign recognition results obtained by manually annotating historical images.

[0190] Optionally, updating the parameters in the feature extraction network and target classification network to be validated can be done by updating the weight matrix and bias matrix, etc., of the feature extraction network and target classification network to be constructed. The weight matrix and bias matrix mentioned above include, but are not limited to, the matrix parameters in the convolutional layers, feedforward layers, and fully connected layers of the feature extraction network and target classification network to be validated.

[0191] In this embodiment, a loss function can be used to calculate the loss value between the results obtained from the validation set input to the classification network to be validated and the label results, thereby updating the parameters in the feature extraction network and the target classification network to be validated. Optionally, the loss function can be the cross-entropy loss function, the normalized cross-entropy loss function, or Focalloss.

[0192] When updating the parameters of the feature extraction network and the target classification network to be validated using the loss function, if the loss function indicates that the feature extraction network and the target classification network to be validated have not converged, the parameters in the model are adjusted to make them converge, thus obtaining the feature extraction network and the target classification network. Convergence of the feature extraction network and the target classification network to be validated can be defined as the difference between the output results of the feature extraction network and the labeled results of the target classification network on the validation set being less than a preset threshold, or the rate of change of the difference between the output results and the labeled results of the training data approaching a certain low value. When the calculated loss function is small, or the difference between it and the loss function output from the previous iteration is close to 0, the feature extraction network and the target classification network to be validated are considered to have converged, and the classification network can be obtained.

[0193] It should be noted that, in the embodiments of this application, the iterative training of the feature extraction network and the target classification network in the classification network are two independent processes. Iterative training can be performed only on the feature extraction network or only on the target classification network. Of course, iterative training can also be performed on both the feature extraction network and the target classification network. The execution order of the two is not limited; they can be executed in sequence in one iteration or in parallel.

[0194] To better understand the embodiments of this application, the complete flowchart of the traffic sign recognition method proposed in this application will be further described below. Figure 13 As shown, the method may include the following steps:

[0195] S501. Perform traffic sign detection on the traffic image to be detected to obtain the traffic sign region image corresponding to the traffic image to be detected.

[0196] Specifically, please see Figure 14 As shown, taking traffic lights as an example, images can be acquired through image acquisition devices or vehicle-mounted photography equipment. Specifically, the road ahead is photographed to obtain the traffic image to be detected. The traffic image to be detected includes not only the traffic sign (traffic light) to be identified, but also background information.

[0197] After acquiring the traffic image to be detected, the computer equipment, considering the limited computing resources, can use lossy compression or lossless compression to compress the directly acquired traffic image to be detected, thus obtaining the processed traffic image to be detected.

[0198] Then, determine the number of anchor frames at each position in the output layer of the light detection network. The width and height of the processed traffic image to be detected can be used as features, and the k-means algorithm is used to cluster the processed traffic image to be detected to obtain multiple clusters. The number of clusters can be the number of anchor frames, and the anchor frame size is determined according to the length and width corresponding to the position coordinates of the center point of each cluster. The trained traffic light detection network performs feature extraction on the processed traffic image to be detected. Specifically, the traffic image to be detected is input into the basic convolutional module and the separable convolutional module in the traffic light detection network. The image passes through the convolutional layer, normalization layer and activation layer in the basic convolutional module in sequence to obtain a feature map. Then, the feature map is convolved by the separable convolutional module to obtain a convolutional feature map. The convolutional feature map is then passed through the convolutional layer, normalization layer and activation layer in the basic convolutional module to obtain the position offset and size offset of the prediction box relative to the anchor box. Based on the position of the anchor box, the size of the anchor box, and the corresponding position offset and size offset, the preset box corresponding to the anchor box is determined. The non-maximum suppression algorithm is used to filter out invalid preset boxes to obtain the traffic light area image.

[0199] S502. Adjust the size of the traffic area image according to the preset size parameters.

[0200] Taking traffic lights as an example, after obtaining the image of the light body area, it can be cropped to obtain the cropped light body area image. Then, preprocessing is performed. By obtaining the position and size information of the light body area image, based on the position and size information of the cropped light body area image, according to the preset size parameters, such as a preset side length s, the light body area image is scaled and resized. After the image scaling and resizing process, the long side is s. For the short side of the light body area image, it can be expanded according to a preset ratio with the traffic light area as the center. For example, it can be expanded by symmetrical padding on both sides, so that the size of the processed light body area image is s*s. Then, according to the mapping relationship between the light body area image and the processed light body area image, the coordinates of the traffic light area are mapped to the processed light body area image of size s*s.

[0201] S503. Determine N anchor boxes in the traffic body region image, input the traffic body region image into the detection network, extract features from the region corresponding to each anchor box, and obtain the position offset and size offset corresponding to each anchor box.

[0202] Specifically, the dimensions of the lamp body region image, including width and height, can be obtained. For each feature point in the lamp body region image, the width and height are used as detection features. Based on these detection features, the k-means algorithm is used to perform clustering, resulting in N clusters. The anchor frame size is determined based on the length and width corresponding to the position coordinates of the center point of each cluster. The center of each of the N clusters is then used as the width and height of the corresponding anchor frame. The size and position of the anchor frames differ for different clusters.

[0203] Please see Figure 15As shown, when the size of the acquired lamp body region image is 144*256, the lamp body region image with 3 input channels and a size of 144*256 is input into the LED detection network. It is processed by the ConvBnReLU module in the LED detection network to obtain a first feature map with 16 output channels and a size of 72*128. Then, the first feature map is processed by the separable convolution module in the LED detection network for feature extraction, resulting in a second feature map with 256 output channels and a size of 18*32. This second feature map with 256 output channels and a size of 18*32 is then further processed by the ConvBnReLU module in the LED detection network, resulting in an output result with 6n output channels and a size of 18*32. This output result includes the position offset (center horizontal coordinate offset and center vertical coordinate offset), size offset (width offset and height offset) corresponding to each type of anchor box, the confidence score of the predicted box, and the predicted category result.

[0204] It should be noted that, to reduce computational load, the aforementioned separable convolution employs a depthwise-pointwise design, and the number of output channels for this LED detection network is N*(4+1+c), where N is the number of defined anchor frames, 4 represents the number of parameters for the positional and dimensional offsets of each anchor frame, where the positional offset can include the offsets of the center horizontal and vertical coordinates, and the dimensional offset can include the width and height offsets. 1 represents the confidence level of the presence of traffic signs in the preset frames, and c represents the number of target categories of whether traffic signs exist in the preset frames. In this embodiment, the categories of traffic signs present in the preset frames are determined, so c is 1. Therefore, the number of output channels for the LED detection network is N*(4+1+1), or 6*N.

[0205] The ConvBnReLU module contains sequentially connected Conv convolutional layers, Bn layers, and ReLU layers. During the processing of the first feature map by the ConvBnReLU module in the LED detection network, image features such as edges and textures within each anchor frame of the lamp body region are extracted using the Conv convolutional layer, resulting in image features of the lamp body region. Noise features are filtered out using the Bn layer, yielding filtered image features. These filtered features are then linearly mapped using a ReLU layer (activation function) to enhance the generalization ability of the feature extraction model, resulting in a first feature map with a size of 72*128. This first feature map includes multiple feature points.

[0206] The aforementioned separable convolutional module can include multiple sequentially connected DW-PW layers. After obtaining a first feature map with an image size of 72*128, it can be input into the first DW-PW layer in the separable module for convolution extraction, resulting in a feature map with 64 output channels and an image size of 36*64. Then, this feature map with 64 output channels and 36*64 image size is passed through the second DW-PW layer in the separable module for convolution extraction, resulting in a feature map with 128 output channels and an image size of 18*32. Similarly, this feature map with 128 output channels and 18*32 image size is passed through the third and fourth DW-PW layers in the separable module for convolution extraction, resulting in a second feature map with 256 output channels and an image size of 18*32. This second feature map includes multiple feature points.

[0207] Optionally, the DW-PW layer mentioned above may also include multiple BN layers and ReLU layers. The BN layer is used to perform normalization processing to obtain normalized features, and the ReLU layer includes an activation function to map its result to the range of 0 to 1.

[0208] S504. Determine the preset frame corresponding to the anchor frame based on the anchor frame's position, anchor frame size, corresponding position offset, and size offset.

[0209] S505. Filter out invalid preset frames in the preset frames corresponding to the N types of anchor frames, and obtain the traffic sign area image based on the area corresponding to the remaining anchor frames.

[0210] Specifically, after obtaining the positional and dimensional offsets of the anchor frames, a preset correction algorithm can be applied to each type of anchor frame. Based on the anchor frame's position, size, and corresponding positional and dimensional offsets, the anchor frames are corrected to obtain N preset frames corresponding to N types of anchor frames. These N preset frames may include frames containing traffic lights or frames without traffic lights. Then, a non-maximum suppression algorithm is used to filter out invalid preset frames from the N preset frames, thereby obtaining preset frames that only contain the traffic light region, thus obtaining the corresponding traffic light region image.

[0211] S506. Cropping the traffic sign area image to obtain the traffic sign image.

[0212] After obtaining the traffic light area image, image processing software can be used to crop the image, removing redundant background areas to obtain the traffic light image. This traffic light image contains only the traffic light area. Then, the traffic light image containing only the traffic light area is adjusted according to different sizes and scaling ratios. The side length s1 of the target size can be defined, and the traffic light area image is adjusted according to a preset scaling ratio so that the size of the traffic light image is s1*s1.

[0213] S507. Input the traffic sign image into the classification network for sign type detection to obtain the pixel information and shape information of the traffic sign image.

[0214] S508. Based on pixel information and shape information, perform semantic prediction to obtain the predicted semantics of the traffic sign image, and determine the traffic sign recognition result of the traffic image to be detected based on the predicted semantics.

[0215] Specifically, please see Figure 16 As shown, after obtaining the traffic light image, and the image size of the traffic light image is 64*64, the traffic light image with 3 input channels and an image size of 64*64 is input into the classification network. It is processed by the ConvBnReLU module in the LED classification network to obtain a third feature map with 16 output channels and an image size of 32*32. Then, the third feature map is processed by the separable convolution module in the LED classification network for feature extraction to obtain a fourth feature map with 256 output channels and an image size of 4*4. The fourth feature map is then processed by the MaxPool pooling module in the LED classification network to obtain a feature map with 256 output channels and an image size of 1*1. Then, it is processed by the fully connected layer FC in the LED classification network to obtain a feature map with C1 output channels and an image size of 1*1. Finally, it is processed by the activation function to obtain the output result, which can be a 1*C1 vector, where the vector value at each position represents the probability value of the result being the corresponding class at that position. Then, the probability value is compared with a preset threshold. The positions with a probability value greater than the preset threshold can be listed as the traffic light prediction results. Suppose that the traffic light recognition results of the traffic image to be detected are "green light for U-turn", "red light for left turn" and "green light for right turn".

[0216] The aforementioned separable convolutional module may include multiple sequentially connected DW-PW layers. After obtaining a third feature map with an image size of 32*32, it can be input into the first DW-PW layer in the separable module for convolution extraction, resulting in a feature map with 64 output channels and an image size of 16*16. Then, this feature map with 64 output channels and an image size of 16*16 is processed through convolution extraction by the second DW-PW layer in the separable module, resulting in a feature map with 128 output channels and an image size of 8*8. Similarly, this feature map with 128 output channels and an image size of 8*8 is processed through convolution extraction by the third DW-PW layer in the separable module, resulting in a fourth feature map with 256 output channels and an image size of 4*4. This fourth feature map includes multiple feature points.

[0217] Furthermore, when autonomous driving assistance is required, the computer device can semantically integrate the traffic light recognition results of "green light for U-turns", "red light for left turns" and "green light for right turns" to obtain semantic prompt information such as "U-turns and right turns are allowed, left turns are prohibited", so that the vehicle can perform the corresponding operation according to the semantic prompt information.

[0218] In this embodiment, traffic sign detection is performed on the traffic image to be detected, thereby obtaining a coarse-grained traffic area image. By determining N anchor boxes in the traffic area image, traffic sign detection can be performed more precisely on the area corresponding to each anchor box. This allows for more granular extraction of image features for each traffic sign area in the traffic area image. Then, by performing sign type detection on the traffic sign area image, the traffic sign recognition result of the traffic image to be detected is obtained by combining fine and comprehensive features, thus improving the accuracy of the traffic sign recognition result.

[0219] It should be noted that although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0220] on the other hand, Figure 17 This is a schematic diagram of a traffic sign recognition device provided in an embodiment of this application. The device can be a component within a terminal or server, such as… Figure 17 As shown, the device 700 includes:

[0221] The traffic body detection module 710 is used to detect traffic signs in the traffic image to be detected, and obtain the traffic body region image corresponding to the traffic image to be detected.

[0222] The sign region detection module 720 is used to determine N anchor frames in the traffic body region image, perform traffic sign detection on the region corresponding to each anchor frame, and obtain the traffic sign region image corresponding to the traffic sign body region image; the size and position of each of the N anchor frames are different, and N is an integer greater than or equal to 1;

[0223] The recognition module 730 is used to perform sign type detection on the traffic sign area image, obtain the type information of traffic signs in the traffic image to be detected, and obtain the traffic sign recognition result of the traffic image to be detected based on the type information.

[0224] In some embodiments, see Figure 18 As shown, the above-mentioned marker area detection module 720 includes:

[0225] The acquisition unit 721 is used to determine N anchor frames in the traffic body area image; each anchor frame has a different size and position, and N is an integer greater than or equal to 1;

[0226] The feature extraction unit 722 is used to input the traffic body region image into the detection network, extract features from the region corresponding to each anchor frame, and obtain the position offset and size offset of each anchor frame in the traffic body region image.

[0227] The determining unit 723 is used to determine the preset frame corresponding to the anchor frame based on the position of the anchor frame, the size of the anchor frame, the position offset of the anchor frame, and the size offset of the anchor frame;

[0228] The filtering unit 724 is used to filter out invalid preset frames in the preset frames corresponding to the N types of anchor frames, and obtain the traffic sign area image based on the area corresponding to the remaining anchor frames.

[0229] In some embodiments, the feature extraction unit 722 is specifically used for:

[0230] Determine the target region of the traffic body region image based on anchor frames;

[0231] The target region is subjected to basic convolution processing to obtain the first feature map; the basic convolution processing includes initial convolution processing, normalization processing and activation function processing in sequence;

[0232] The first feature map is subjected to separable convolution to obtain the second feature map;

[0233] The second feature map is subjected to basic convolution processing to obtain the position offset and size offset.

[0234] In some embodiments, the above-described apparatus is further configured to:

[0235] The image of the traffic area is resized according to preset size parameters.

[0236] In some embodiments, the above-described apparatus is further configured to:

[0237] Acquire the location and size information of the traffic volume area image;

[0238] Based on the location and size information of the traffic body area image, the traffic body area image is scaled and expanded according to preset size parameters, and the coordinate position of the traffic body area is mapped.

[0239] In some embodiments, the training process of the detection network includes:

[0240] Obtain M anchor boxes from the historical traffic volume image; each anchor box has a different size and position, M is an integer greater than or equal to 1, and the historical traffic volume image contains the labeled traffic volume region;

[0241] Input the historical traffic body image into the first initial network, extract features from the historical traffic body image, and obtain the position offset and size offset of each anchor frame in M ​​anchor frames;

[0242] For each of the M anchor frames, the prediction frame corresponding to the anchor frame is determined based on the anchor frame's position, size, position offset, and size offset.

[0243] Invalid prediction frames are filtered out from the prediction frames corresponding to the M anchor frames to obtain the prediction traffic volume region of the first initial network;

[0244] Based on the loss function between the predicted traffic body region and the labeled traffic body region, the first initial network is iteratively trained using an iterative algorithm to minimize the loss function, thereby obtaining the detection network.

[0245] In some embodiments, the loss function includes a first component, a second component, a third component, and a fourth component;

[0246] The first component is used to characterize the positional offset loss between the prediction box and the anchor box;

[0247] The second component is used to characterize the loss of dimensional offset between the prediction box and the anchor box;

[0248] The third component is used to characterize the loss between the predicted confidence level and the true confidence level corresponding to the predicted box;

[0249] The fourth component is used to characterize the loss between the predicted category result and the true category result corresponding to the prediction box.

[0250] In some embodiments, the identification module 730 is specifically used for:

[0251] The traffic sign area image is cropped to obtain the traffic sign image;

[0252] Traffic sign images are input into a classification network for sign type detection to obtain pixel and shape information of the traffic sign images;

[0253] Semantic prediction is performed based on pixel and shape information to obtain the predicted semantics of traffic sign images, and the traffic sign recognition result of the traffic image to be detected is determined based on the predicted semantics.

[0254] In some embodiments, the training process of the classification network includes:

[0255] Historical traffic sign area images are acquired, and historical traffic sign images are determined based on each historical traffic sign area image; each historical traffic sign image is labeled with the corresponding historical classification result.

[0256] Each classification network is trained based on each historical traffic sign image and each historical classification result.

[0257] In some embodiments, each device is further configured to:

[0258] Historical traffic sign images are input into a second initial network for sign type detection to obtain pixel and shape information of the historical traffic sign images;

[0259] Semantic prediction is performed based on pixel and shape information of historical traffic sign images to obtain the predicted semantics of historical traffic sign images, and the prediction result of historical traffic sign images is determined based on the predicted semantics.

[0260] Based on the prediction results and historical classification results of historical traffic sign images, a loss function is calculated. By minimizing the loss function, an iterative algorithm is used to adjust the parameters of the second initial network to obtain the classification network.

[0261] In some embodiments, the above-described apparatus is further configured to:

[0262] The historical traffic sign area image is resized and data augmented to obtain the historical traffic sign image.

[0263] It is understood that the functions of each functional module of the traffic sign recognition device in this embodiment can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the above method embodiments, and will not be repeated here.

[0264] In summary, the cell data annotation device provided in this application, on the one hand, after obtaining the guidance information of the traffic body region image, can perform traffic sign detection more precisely on the region corresponding to each anchor frame by determining N anchor frames in the traffic body region image. This allows for more granular extraction of image features of each traffic sign region in the traffic body region image, enabling the identification of traffic signs in the image based on more detailed features, thus effectively improving the accuracy of traffic sign recognition. On the other hand, by performing sign type detection on the traffic sign region image, more refined and comprehensive features can be combined to determine the traffic sign recognition result of the traffic image to be detected, which also significantly improves the recognition accuracy of the method provided in this application compared to the prior art.

[0265] On the other hand, the device provided in the embodiments of this application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the traffic sign recognition method as described above.

[0266] The following is for reference. Figure 19 , Figure 19 This is a schematic diagram of the computer system structure of the terminal device according to an embodiment of this application.

[0267] like Figure 19 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 303 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the system 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0268] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0269] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 303, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined above in the system of this application.

[0270] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0271] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0272] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, it can be described as: a processor including: a traffic body detection module, a sign area detection module, and a recognition module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the traffic body detection module can also be described as "used to detect traffic signs in a traffic image to be detected, and obtain a traffic body area image corresponding to the traffic image to be detected."

[0273] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs, which, when used by one or more processors, execute the traffic sign recognition method described in this application:

[0274] Traffic sign detection is performed on the traffic image to be detected to obtain the traffic sign region image corresponding to the traffic image to be detected.

[0275] N anchor frames are determined in the traffic body region image. Traffic sign detection is performed on the region corresponding to each anchor frame to obtain the traffic sign region image corresponding to the traffic sign body region image. The size and position of each of the N anchor frames are different, and N is an integer greater than or equal to 1.

[0276] The traffic sign area image is subjected to sign type detection to obtain the type information of traffic signs in the traffic image to be detected, and the traffic sign recognition result of the traffic image to be detected is obtained based on the type information.

[0277] In summary, the traffic sign recognition method, apparatus, device, and medium provided in this application embodiment detect traffic sign bodies in a traffic image to be detected, obtaining a traffic body region image corresponding to the traffic image to be detected. Then, N anchor frames are determined in the traffic body region image, and traffic sign detection is performed on the region corresponding to each anchor frame to obtain a traffic sign region image corresponding to the traffic sign body region image. Furthermore, sign type detection is performed on the traffic sign region image to obtain the type information of the traffic signs in the traffic image to be detected. Based on the type information, the traffic sign recognition result of the traffic image to be detected is obtained. Compared with the prior art, the technical solution in this application, on the one hand, by determining N anchor frames in the traffic body region image after obtaining the guiding information of the traffic body region image, allows for more precise traffic sign detection on the region corresponding to each anchor frame. This enables more granular extraction of image features for each traffic sign region in the traffic body region image, allowing for more detailed feature recognition of traffic signs in the image, effectively improving the accuracy of traffic sign recognition. On the other hand, by performing sign type detection on the traffic sign area image, more refined and comprehensive features can be combined to determine the traffic sign recognition result of the traffic image to be detected, and the recognition accuracy of the method provided in this application can be significantly improved compared with the prior art to a certain extent.

[0278] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A traffic sign recognition method, characterized in that, include: Traffic sign detection is performed on the traffic image to be detected to obtain the traffic sign region image corresponding to the traffic image to be detected. N anchor boxes are determined in the traffic body region image. The traffic body region image is input into the detection network. The target region of the traffic body region image is determined based on the anchor boxes. The target region is subjected to basic convolution processing to obtain the first feature map. The first feature map is subjected to separable convolution to obtain the second feature map; The second feature map is subjected to the basic convolution process to obtain the position offset and size offset of each anchor frame in the traffic body region image; The preset frame corresponding to the anchor frame is determined based on the position of the anchor frame, the size of the anchor frame, the position offset of the anchor frame, and the size offset of the anchor frame; Invalid preset frames are filtered out from the preset frames corresponding to the N anchor frames, and each traffic sign area image corresponding to the traffic body area image is obtained based on the area corresponding to the remaining anchor frames; the size and position of each anchor frame in the N anchor frames are different, and N is an integer greater than or equal to 1; the invalid preset frames include preset frames that do not completely contain the traffic sign area and preset frames that contain the background area; Each traffic sign area image is input into a trained classification network for sign type detection to obtain the type information of each traffic sign in the traffic image to be detected. Based on the type information, the traffic sign recognition result of the traffic image to be detected is obtained. The traffic sign recognition result includes multiple type attributes under the traffic sign type. The training process of the classification network includes: Historical traffic sign area images are acquired, and historical traffic sign images are determined based on the historical traffic sign area images; the historical traffic sign images are labeled with corresponding historical classification results; in the historical traffic sign images, traffic signs of rare categories are obtained by data augmentation using traffic signs of other types of the same color; the other types are types other than the traffic sign types corresponding to the historical traffic sign images of rare categories; The classification network is trained based on the historical traffic sign images and the historical classification results; The training process of the detection network includes: M anchor boxes are obtained from a historical traffic body image; each anchor box has a different size and position, where M is an integer greater than or equal to 1, and the historical traffic body image contains labeled traffic body regions; the historical traffic body image is input into a first initial network to extract features from it, obtaining the position offset and size offset corresponding to each of the M anchor boxes; for each of the M anchor boxes, a prediction box corresponding to the anchor box is determined based on the anchor box's position, size, and corresponding position and size offsets; invalid prediction boxes in the prediction boxes corresponding to the M anchor boxes are filtered out to obtain the predicted traffic body regions of the first initial network; based on the loss function between the predicted traffic body regions and the labeled traffic body regions, the first initial network is iteratively trained using an iterative algorithm to minimize the loss function, thereby obtaining a detection network.

2. The method according to claim 1, characterized in that, The basic convolution processing includes initial convolution processing, normalization processing, and activation function processing in sequence.

3. The method according to claim 1, characterized in that, Before performing traffic sign detection on the traffic body region image, the method further includes: The image of the traffic body area is resized according to preset size parameters.

4. The method according to claim 3, characterized in that, The image of the traffic body area is resized according to preset size parameters, including: Obtain the location and size information of the traffic body area image; Based on the location and size information of the traffic body area image, the traffic body area image is scaled and expanded according to preset size parameters, and the coordinate position of the traffic body area is mapped.

5. The method according to claim 1, characterized in that, The loss function includes a first component, a second component, a third component, and a fourth component; The first component is used to characterize the positional offset loss between the predicted frame and the anchor frame; The second component is used to characterize the loss of dimensional offset between the predicted frame and the anchor frame; The third component is used to characterize the loss between the predicted confidence level and the true confidence level corresponding to the prediction box; The fourth component is used to characterize the loss between the predicted category result and the true category result corresponding to the prediction box.

6. The method according to claim 1, characterized in that, The method further includes: The traffic sign area image is cropped to obtain the traffic sign image; The traffic sign image is input into a classification network for sign type detection to obtain the pixel information and shape information of the traffic sign image; Semantic prediction is performed based on the pixel information and the shape information to obtain the predicted semantics of the traffic sign image, and the traffic sign recognition result of the traffic image to be detected is determined based on the predicted semantics.

7. The method according to claim 1, characterized in that, Training the classification network based on the historical traffic sign images and the historical classification results includes: The historical traffic sign image is input into the second initial network for sign type detection to obtain the pixel information and shape information of the historical traffic sign image; Semantic prediction is performed based on the pixel information and shape information of the historical traffic sign image to obtain the predicted semantics of the historical traffic sign image, and the prediction result of the historical traffic sign image is determined based on the predicted semantics. Based on the prediction results and historical classification results of the historical traffic sign images, a loss function is calculated. By minimizing the loss function, the parameters of the second initial network are iteratively adjusted using an iterative algorithm to obtain the classification network.

8. The method according to claim 1, characterized in that, Determining historical traffic sign images based on the historical traffic sign area images includes: The historical traffic sign area image is resized and data augmented to obtain a historical traffic sign image.

9. A traffic sign recognition device, characterized in that, The device includes: The traffic body detection module is used to detect traffic signs in the traffic image to be detected, and to obtain the traffic body region image corresponding to the traffic image to be detected. A sign region detection module is used to determine N anchor boxes in the traffic body region image. The traffic body region image is input into a detection network. Based on the anchor boxes, a target region of the traffic body region image is determined. Basic convolution processing is performed on the target region to obtain a first feature map. Separable convolution processing is performed on the first feature map to obtain a second feature map. Basic convolution processing is performed on the second feature map to obtain the position offset and size offset corresponding to each anchor box in the traffic body region image. Based on the position, size, and position and size offset of the anchor box, a preset box corresponding to the anchor box is determined. Invalid preset boxes among the N preset boxes are filtered out. Based on the remaining anchor box regions, each traffic sign region image corresponding to the traffic body region image is obtained. The size and position of each of the N anchor boxes are different, where N is an integer greater than or equal to 1. Invalid preset boxes include preset boxes that do not completely contain the traffic sign region and preset boxes that contain the background region. The recognition module is used to input each traffic sign area image into a trained classification network for sign type detection, obtain type information of each traffic sign in the traffic image to be detected, and obtain the traffic sign recognition result of the traffic image to be detected based on the type information. The traffic sign recognition result includes multiple type attributes under the traffic sign type. The training process of the classification network includes: acquiring historical traffic sign area images; determining historical traffic sign images based on the historical traffic sign area images; the historical traffic sign images are labeled with corresponding historical classification results; in the historical traffic sign images, rare categories of traffic signs are obtained by data augmentation using other types of traffic signs of the same color; the other types are types other than the traffic sign types corresponding to the rare categories of historical traffic sign images; and training the classification network based on the historical traffic sign images and the historical classification results. The training process of the detection network includes: obtaining M anchor boxes from a historical traffic body image; each anchor box has a different size and position, where M is an integer greater than or equal to 1, and the historical traffic body image contains labeled traffic body regions; inputting the historical traffic body image into a first initial network to extract features from the historical traffic body image, obtaining the position offset and size offset corresponding to each of the M anchor boxes; for each of the M anchor boxes, determining the prediction box corresponding to the anchor box based on the anchor box's position, size, and corresponding position offset and size offset; filtering out invalid prediction boxes in the prediction boxes corresponding to the M anchor boxes to obtain the predicted traffic body regions of the first initial network; and iteratively training the first initial network using an iterative algorithm based on the loss function between the predicted traffic body regions and the labeled traffic body regions, minimizing the loss function, to obtain the detection network.

10. The apparatus according to claim 9, characterized in that, The basic convolution processing includes initial convolution processing, normalization processing, and activation function processing in sequence.

11. The apparatus according to claim 9, characterized in that, The device is also used to perform size adjustment processing on the traffic body area image according to preset size parameters.

12. The apparatus according to claim 11, characterized in that, The device is also used to acquire the location and size information of the traffic body area image; based on the location and size information of the traffic body area image, perform image scaling and expansion processing on the traffic body area image according to preset size parameters, and map the coordinate position of the traffic body area.

13. The apparatus according to claim 9, characterized in that, The loss function includes a first component, a second component, a third component, and a fourth component; The first component is used to characterize the positional offset loss between the predicted frame and the anchor frame; The second component is used to characterize the loss of dimensional offset between the predicted frame and the anchor frame; The third component is used to characterize the loss between the predicted confidence level and the true confidence level corresponding to the prediction box; The fourth component is used to characterize the loss between the predicted category result and the true category result corresponding to the prediction box.

14. The apparatus according to claim 9, characterized in that, The recognition module is used to crop the traffic sign area image to obtain a traffic sign image; input the traffic sign image into a classification network for sign type detection to obtain pixel information and shape information of the traffic sign image; perform semantic prediction based on the pixel information and shape information to obtain the predicted semantics of the traffic sign image; and determine the traffic sign recognition result of the traffic image to be detected based on the predicted semantics.

15. The apparatus according to claim 9, characterized in that, The device is further configured to input the historical traffic sign image into a second initial network for sign type detection, thereby obtaining pixel information and shape information of the historical traffic sign image; perform semantic prediction based on the pixel information and shape information of the historical traffic sign image to obtain the predicted semantics of the historical traffic sign image; determine the prediction result of the historical traffic sign image based on the predicted semantics; calculate a loss function based on the prediction result of the historical traffic sign image and the historical classification result; minimize the loss function; and iteratively adjust the parameters of the second initial network using an iterative algorithm to obtain the classification network.

16. The apparatus according to claim 9, characterized in that, The device is also used to resize and enhance the data of the historical traffic sign area image to obtain a historical traffic sign image.

17. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the traffic sign recognition method as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, It stores a computer program for implementing the traffic sign recognition method as described in any one of claims 1-8.

19. A computer program product, characterized in that, The computer program product includes instructions that, when executed, implement the traffic sign recognition method as described in any one of claims 1-8.

Images