A roadside sign recognition method, device and electronic equipment

Abstract

The application discloses a roadside sign recognition method, device and electronic equipment. The roadside sign recognition method comprises the following steps: acquiring an input image comprising a roadside sign; performing feature extraction on the input image by using a convolutional neural network to obtain a multi-scale feature map of the input image; when performing feature fusion on the multi-scale feature map according to at least two branch structures, obtaining a fusion feature map corresponding to each branch structure, and the feature map scales corresponding to each branch structure are different; and obtaining an output image corresponding to the input image according to the fusion feature map corresponding to each branch structure, wherein the output image comprises a recognition position of the roadside sign. The technical scheme can improve the recognition accuracy of various roadside signs, including a road sign corner point.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a roadside sign recognition method, device, and electronic device. Background Technology

[0002] In the fields of assisted driving and autonomous driving research, perceiving the vehicle's current environment is a prerequisite for achieving automatic control. Road sign information (such as speed limit signs) is crucial traffic information; failure to accurately obtain this information can lead to traffic violations and safety hazards. Accurately identifying road sign content provides vital data support for technologies such as navigation, assisted driving, and autonomous driving, preventing violations and safety risks.

[0003] The core prerequisite for recognizing road sign content is the ability to accurately identify the corner points of the road sign from the collected image data, and then determine the road sign content based on the identified corner points. However, current technologies suffer from low accuracy in recognizing road sign corner points. Summary of the Invention

[0004] In view of this, the main objective of this application is to provide a roadside sign recognition method, device and electronic device that can accurately identify roadside signs from image data.

[0005] According to a first aspect of this application, a roadside sign recognition method is provided, comprising:

[0006] Acquire an input image including roadside signs;

[0007] A convolutional neural network is used to extract features from the input image to obtain a multi-scale feature map of the input image;

[0008] When the multi-scale feature map is fused according to at least two branch structures, a fused feature map corresponding to each branch structure is obtained, and the feature map scale corresponding to each branch structure is different.

[0009] The output image corresponding to the input image is obtained by identifying the fusion feature map corresponding to each branch structure, and the output image includes the identification location of the roadside sign.

[0010] In some embodiments, the feature map scales corresponding to at least two branch structures are set through the following steps:

[0011] The maximum scale in the multi-scale feature map is obtained as the feature map scale corresponding to one of the at least two branch structures, and the feature map scale corresponding to the other branch structures in the at least two branch structures is set to be greater than the maximum scale in the multi-scale feature map.

[0012] In some embodiments, when the multi-scale feature map is fused according to at least two branch structures, a fused feature map corresponding to each branch structure is obtained, including:

[0013] Based on the feature map scale corresponding to each branch structure, the multi-scale feature map is scaled and transformed to obtain a feature map to be fused with the same scale as the feature map corresponding to each branch structure.

[0014] The feature maps to be fused corresponding to each branch structure are superimposed to obtain the initial fused feature map corresponding to each branch structure.

[0015] Based on the feature map scale corresponding to each branch structure, the initial fusion feature map corresponding to the smaller-scale branch structure is superimposed onto the initial fusion feature map corresponding to the larger-scale branch structure to obtain the fusion feature map corresponding to the larger-scale branch structure.

[0016] In some embodiments, based on the feature map scale corresponding to each branch structure, the initial fused feature map corresponding to the smaller-scale branch structure is superimposed onto the initial fused feature map corresponding to the larger-scale branch structure to obtain the fused feature map corresponding to the larger-scale branch structure, including:

[0017] The initial fusion feature map corresponding to the smaller-scale branch structure is convolved, and the scale of the initial fusion feature map after convolution is transformed so that the scale of the scale-transformed initial fusion feature map is the same as the scale of the feature map corresponding to the larger-scale branch structure.

[0018] The initial fusion feature map after scaling the smaller-scale branch structure is linearly superimposed onto the initial fusion feature map corresponding to the larger-scale branch structure to obtain the fusion feature map corresponding to the larger-scale branch structure.

[0019] In some embodiments, the multi-scale feature maps are scale-transformed according to the feature map scale corresponding to each branch structure to obtain a feature map to be fused with the same feature map scale as the feature map corresponding to each branch structure, including:

[0020] Each scale feature map is compared with the scale of the feature map corresponding to each branch structure. If the scale of the scale feature map is larger than the scale of the feature map corresponding to the branch structure, the scale feature map is upsampled to obtain a feature map to be fused with the same scale as the feature map corresponding to the branch structure.

[0021] If the scale of the scale feature map is equal to the scale of the feature map corresponding to the branch structure, the scale feature map is directly used as the feature map to be fused corresponding to the branch structure.

[0022] In some embodiments, the output image corresponding to the input image is obtained based on the fusion feature map corresponding to each branch, including:

[0023] Based on the scale of the input image, the scale transformation of the fusion feature maps corresponding to each branch structure is performed so that the scale of each fusion feature map after scale transformation is the same as the scale of the input image.

[0024] The output image corresponding to the input image is obtained based on the fused feature map after scale transformation corresponding to each branch.

[0025] In some embodiments, the output image corresponding to the input image is obtained based on the scale-transformed fused feature map corresponding to each branch, including:

[0026] The scale-transformed fused feature maps corresponding to each branch are linearly superimposed according to pixel positions to obtain the output image.

[0027] According to a second aspect of this application, a roadside sign recognition device is provided, comprising:

[0028] An acquisition unit is used to acquire an input image including roadside signs;

[0029] The computing unit is used to extract features from the input image using a convolutional neural network to obtain a multi-scale feature map of the input image.

[0030] The fusion unit is used to obtain a fused feature map corresponding to each branch structure when performing feature fusion on the multi-scale feature map according to at least two branch structures, and the feature map scale corresponding to each branch structure is different.

[0031] The recognition unit is used to identify the output image corresponding to the input image based on the fusion feature map corresponding to each branch structure, and the output image includes the recognition location of the roadside sign.

[0032] According to a third aspect of this application, an electronic device is provided, including a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the roadside sign recognition method of the above embodiments.

[0033] According to a fourth aspect of this application, a computer-readable storage medium is provided that stores one or more programs, which, when executed by a processor, implement the roadside sign recognition method of the above embodiments.

[0034] The above-mentioned technical solutions adopted in this application can achieve the following beneficial effects: The roadside sign recognition method, device and electronic device of this application use convolutional neural networks to calculate multi-scale feature maps of input images. When performing feature fusion on the multi-scale feature maps, feature fusion is performed according to at least two branch structures with different scales. Since the scale information of the fused feature maps corresponding to each branch structure is different, the feature information lost in the feature fusion process is also different, ensuring that the fused feature maps corresponding to each branch structure retain more feature information in general. In this way, when classifying and recognizing the input image according to the fused feature maps corresponding to each branch structure to obtain the output image, the recognition accuracy of the position of each feature element in the input image can be improved.

[0035] Furthermore, the embodiments of this application increase the number of branches in the feature fusion stage to improve recognition accuracy. The amount of added parameters is minimal. Compared with the prior art, the recognition accuracy of this application is higher while having almost the same recognition efficiency. Attached Figure Description

[0036] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0037] Figure 1 A flowchart of a roadside sign recognition method according to an embodiment of this application is shown;

[0038] Figure 2 A schematic diagram of feature map fusion in a prior art is shown;

[0039] Figure 3 A schematic diagram of feature map fusion according to one embodiment of this application is shown;

[0040] Figure 4 A schematic diagram of an output image acquisition process according to an embodiment of this application is shown;

[0041] Figure 5 A schematic diagram of a roadside sign recognition device according to an embodiment of this application is shown;

[0042] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0043] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. These embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein.

[0044] Figure 1 A schematic flowchart of a roadside sign recognition method according to an embodiment of this application is shown, as follows: Figure 1 As shown, the method in this application embodiment includes at least steps S110 to S140:

[0045] Step S110: Obtain an input image including roadside signs.

[0046] The roadside sign recognition method of this application embodiment can be implemented by a roadside unit and / or an on-board computing platform and / or a cloud computing platform. Taking the field of autonomous driving as an example, the roadside sign recognition method of this application embodiment can be implemented by an on-board computing platform. The input image can be an image captured by an on-board camera, and the image captured by the on-board camera is used as the input image to be recognized. Of course, the input image can also come from the cloud or other connected vehicles; this embodiment does not limit the source of the input image. Optionally, the input image in this embodiment is an image with a standard scale after scaling.

[0047] The input image in this embodiment includes roadside markings, which include one or more feature elements. Roadside markings include, for example, road traffic signs and road markings. Taking a road sign as an example, a road sign is a traffic instruction sign installed on the side of a lane, and it contains relevant traffic information. Road signs are usually square in shape, and a square road sign has four right-angle points, which are called road sign corner points. These four road sign corner points can be understood as four feature elements.

[0048] This application embodiment performs a series of processes on the input image to finally obtain the location of roadside signs in the input image, such as the location of each feature element of the roadside sign. The number of roadside signs included in the input image is not limited to one; it may be multiple.

[0049] Step S120: Use a convolutional neural network to extract features from the input image to obtain a multi-scale feature map of the input image.

[0050] Convolutional Neural Networks (CNNs) are deep, feedforward neural networks used for image processing. They involve convolutional computations and have a specific structure. A CNN typically consists of multiple base layers, generally including convolutional layers and pooling layers. Convolutional layers perform convolutional computations and feature extraction, while pooling layers reduce the dimensionality of the input image and extract key features. After multi-scale convolutional processing, the input image yields multi-scale feature maps. For example... Figure 3 As shown, after processing the input image through convolution and pooling layers, feature maps of three scales are obtained, each scale generally including multiple channel feature maps.

[0051] It should be noted that the scale information of the multi-scale feature map in this application is related to the scale of the convolutional layer. Different scales can represent the features of the input image from different dimensions. Those skilled in the art can flexibly set the scales of the convolutional layer according to actual needs.

[0052] Step S130: When performing feature fusion on the multi-scale feature map according to at least two branch structures, a fused feature map corresponding to each branch structure is obtained, and the feature map scale corresponding to each branch structure is different.

[0053] like Figure 2 As shown, in existing technologies that use convolutional neural networks to predict the location of feature elements in an image, the feature element locations are calculated using the same feature map. For example... Figure 2 As shown, feature maps at three scales, such as 1 / 4 scale, 1 / 8 scale, and 1 / 16 scale feature maps, are scaled separately before feature map fusion, resulting in, for example... Figure 2 The first half-scale fusion feature map is obtained by convolving the first half-scale fusion feature map with the second half-scale fusion feature map, and then superimposing the second half-scale fusion feature map with the first half-scale fusion feature map. This second half-scale fusion feature map is used to identify the location of feature elements. It is evident that in existing technologies, the location of feature elements in the input image is calculated based on this second half-scale fusion feature map, which is derived from the first half-scale fusion feature map. During the calculation of the first half-scale fusion feature map, some feature information is inevitably lost during the scaling transformation of the first half-scale and first half-scale feature maps. Consequently, the final second half-scale fusion feature map also inevitably loses this part of the feature information, resulting in limited recognition accuracy.

[0054] It should be noted that the scales of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 in the embodiments of this application refer to the feature map size at the 1 / M scale being 1 / M of the size of the input image, where M = 2. n n is a natural number greater than or equal to 1.

[0055] To address this issue, this application embodiment fuses multi-scale feature maps according to at least two branch structures. This results in fused feature maps corresponding to at least two branch structures. Since the feature maps corresponding to each branch structure have different scales, the feature information lost during multi-scale feature map fusion varies due to scale transformation. Therefore, the fused feature maps corresponding to at least two branch structures lose different feature information. The fused feature maps complement each other, resulting in richer and more comprehensive feature information retained overall. This improves the recognition accuracy when identifying feature elements in the input image based on the fused feature maps corresponding to at least two branch structures.

[0056] Step S140: Based on the fusion feature map corresponding to each branch structure, the output image corresponding to the input image is obtained, and the output image includes the identification position of the roadside sign.

[0057] This embodiment superimposes and fuses the fusion feature maps corresponding to each branch structure to obtain the final fusion feature map. The feature elements of the final fusion feature map are then classified and identified to obtain a classification value corresponding to each pixel position. Based on this classification value, the position of each feature element in the input image can be determined. When the roadside marker is a road sign, this embodiment can accurately identify the positions of the four road sign corner points.

[0058] visible, Figure 1 The method shown in this embodiment utilizes a convolutional neural network to calculate multi-scale feature maps of the input image. When fusing features from the multi-scale feature maps, feature fusion is performed according to at least two branch structures with different scales. Since the scale information of the fused feature maps corresponding to each branch structure is different, the feature information lost during feature fusion is also different, ensuring that the fused feature maps corresponding to each branch structure retain more feature information overall. In this way, when classifying and recognizing the input image based on the fused feature maps corresponding to each branch structure to obtain the output image, the recognition accuracy of the position of each feature element in the input image can be improved. Moreover, this embodiment increases the number of branches in the feature fusion stage to improve recognition accuracy, and the amount of added parameters is minimal. Compared with the prior art, this application has higher recognition accuracy while having almost the same recognition efficiency.

[0059] The embodiments of this application determine at least two branch structures through the following steps:

[0060] The maximum scale in the multi-scale feature map is obtained as the feature map scale corresponding to one of the at least two branch structures, and the feature map scale corresponding to the other branch structures in the at least two branch structures is set to be greater than the maximum scale in the multi-scale feature map.

[0061] In other words, when two branch structures are set in the embodiments of this application, the scale of one of the branch structures is the scale of the largest scale feature map in the multi-scale feature map, and the scale of the other branch structure is greater than the scale of the largest scale feature map in the multi-scale feature map.

[0062] like Figure 3 As shown, the multi-scale feature maps obtained by convolution of the input image include, for example, 1 / 4 scale feature maps, 1 / 8 scale feature maps, and 1 / 16 scale feature maps. Figure 3 If there is a feature map of a certain scale, then two branch structures are set accordingly. The feature map scale corresponding to the first branch structure is 1 / 4, and the feature map scale corresponding to the second branch structure is 1 / 2. Of course, the feature map scale corresponding to the second branch structure can also be 1 / 1.

[0063] It should be noted that in some optional embodiments of this example, two or three branch structures are set so as to increase the number of parameters by a very small amount while retaining more feature information, without affecting the recognition efficiency of the input image.

[0064] In some embodiments of this application, when performing feature fusion on a multi-scale feature map according to at least two-branch structures, a fused feature map corresponding to each branch structure is obtained, including:

[0065] First, based on the feature map scale corresponding to each branch structure, the multi-scale feature map is scaled to obtain a feature map to be fused with the same scale as the feature map corresponding to each branch structure.

[0066] Specifically, the feature maps at each scale are compared with the feature map scales corresponding to each branch structure. As shown in the previous example of setting the feature map scales for branch structures, the scale of the multi-scale feature maps is either equal to or smaller than the feature map scale corresponding to the branch structure. Therefore, if the scale of the scale feature map is larger than the feature map scale corresponding to the branch structure, the scale feature map is upsampled to obtain a feature map to be fused with the same scale as the feature map scale corresponding to the branch structure; if the scale of the scale feature map is equal to the feature map scale corresponding to the branch structure, the scale feature map is directly used as the feature map to be fused corresponding to the branch structure.

[0067] Combination Figure 3As shown, for a 1 / 4 branch structure, the 1 / 4 scale feature map can be directly used as the feature map to be fused, while the 1 / 8 and 1 / 16 scale feature maps need to be upsampled before they can be used as the feature maps to be fused. Similarly, for a 1 / 2 branch structure, the 1 / 4, 1 / 8, and 1 / 16 scale feature maps all need to be upsampled before they can be used as the feature maps to be fused.

[0068] Next, after obtaining the feature maps to be fused corresponding to each branch structure, the feature maps to be fused corresponding to each branch structure are superimposed to obtain the initial fused feature map corresponding to each branch structure.

[0069] Specifically, in this embodiment, the feature maps to be fused corresponding to each branch structure are linearly superimposed, for example, by... Figure 3 The feature maps to be fused in the three branches after mesoscale transformation are linearly superimposed to obtain the initial fused feature maps corresponding to the 1 / 4 branch structure and the 1 / 2 branch structure, respectively.

[0070] Finally, after obtaining the initial fusion feature maps corresponding to each branch structure, the initial fusion feature maps corresponding to the smaller-scale branch structure are superimposed onto the initial fusion feature maps corresponding to the larger-scale branch structure according to the feature map scale of each branch structure, to obtain the fusion feature maps corresponding to the larger-scale branch structure.

[0071] Specifically, the initial fusion feature map corresponding to the smaller-scale branch structure is convolved, and the scale of the convolved initial fusion feature map is scaled so that the scale of the scaled initial fusion feature map is the same as that of the feature map corresponding to the larger-scale branch structure. The scaled initial fusion feature map of the smaller-scale branch structure is linearly superimposed on the initial fusion feature map corresponding to the larger-scale branch structure to obtain the fusion feature map corresponding to the larger-scale branch structure.

[0072] by Figure 3 Taking the example shown, after convolution processing the initial fusion feature map corresponding to the 1 / 4 branch structure, the fusion feature map corresponding to the 1 / 4 branch structure is obtained. By linearly superimposing the fusion feature map corresponding to the 1 / 4 branch structure onto the initial fusion feature map corresponding to the 1 / 2 branch structure, the fusion feature map corresponding to the 1 / 2 branch structure is obtained. In this way, the fusion feature maps corresponding to each branch structure are obtained.

[0073] It should be understood that this embodiment performs convolution calculations after each linear superposition to prevent the introduction of noise.

[0074] In some embodiments of this application, the output image corresponding to the input image is obtained based on the fusion feature map corresponding to each branch, including:

[0075] Based on the size of the input image, the fusion feature maps corresponding to each branch structure are scaled so that the size of each fusion feature map after scale transformation is the same as the size of the input image.

[0076] For example, if the input image size is 12*12, the fusion feature map corresponding to the 1 / 4 branch structure has a size of 3*3, and the fusion feature map corresponding to the 1 / 2 branch structure has a size of 6*6, then it is necessary to perform a scaling transformation on the fusion feature map corresponding to the 1 / 4 branch structure to change its size to 12*12, and also perform a scaling transformation on the fusion feature map corresponding to the 1 / 2 branch structure to change its size to 12*12. The scaling transformation method here could be, for example, upsampling the fusion feature maps corresponding to each branch structure.

[0077] After scaling the fusion feature maps corresponding to each branch structure, the output image corresponding to the input image is obtained based on the scale-scaled fusion feature map corresponding to each branch.

[0078] Since this embodiment also classifies and identifies the fusion feature maps corresponding to each branch structure to obtain the classification value corresponding to each pixel position when obtaining the output image, this embodiment linearly superimposes the scale-transformed fusion feature maps corresponding to each branch according to the pixel position to obtain the output image.

[0079] In other words, in this embodiment, the classification values ​​of the same pixel position in the scale-transformed fused feature map corresponding to each branch structure are linearly added together to obtain the final classification value of the pixel position. The classification value indicates the probability that the pixel position corresponds to that category. The larger the value, the more likely the pixel position is to belong to that category.

[0080] Specifically, in combination Figure 4 As shown, after transforming the size of the fused feature map corresponding to the 1 / 4 branch structure and the fused feature map corresponding to the 1 / 2 branch structure to 12*12, the classification values ​​of corresponding pixel positions in the two 12*12 fused feature maps are added together. For example, starting from the top left pixel position of the fused feature map, the classification values ​​are added pixel by pixel. Figure 4Taking two pixel positions as an example, the classification values ​​corresponding to the top-left pixel in the two fused feature maps are 0.1 and 0.01 respectively. After adding them together, the classification value corresponding to the top-left pixel is determined to be 0.11. Similarly, the classification values ​​corresponding to the adjacent pixels in the top-left corner are 0.05 and 0.03 respectively. After adding them together, the classification value corresponding to the adjacent pixels in the top-left corner is determined to be 0.08. In this way, the final classification value of each pixel position in the 12*12 size can be obtained, the output image is obtained, and the entire recognition process is completed.

[0081] In summary, the roadside sign recognition method of this application has the following effects:

[0082] In the feature fusion stage, at least two fusion feature maps corresponding to branch structures at different scales are obtained. The different scale information of the fusion feature maps corresponding to each branch structure leads to different loss of feature information during the feature fusion process, so that the fusion feature maps corresponding to each branch structure generally retain more feature information. In this way, when classifying and recognizing the input image based on the fusion feature maps corresponding to each branch structure to obtain the output image, the recognition accuracy of the position of each feature element in the input image can be improved.

[0083] Furthermore, the embodiments of this application increase the number of branches in the feature fusion stage to improve recognition accuracy. The amount of added parameters is minimal. Compared with the prior art, the recognition accuracy of this application is higher while having almost the same recognition efficiency.

[0084] Similar to the roadside sign recognition method in the foregoing embodiments, this application also provides a roadside sign recognition device for implementing the roadside sign recognition method in the foregoing embodiments.

[0085] Figure 5 A schematic diagram of the structure of a roadside sign recognition device according to an embodiment of this application is shown, such as... Figure 5 As shown, the roadside sign recognition device 500 includes: an acquisition unit 510, a calculation unit 520, a fusion unit 530, and a recognition unit 540;

[0086] The acquisition unit 510 is used to acquire an input image including roadside signs;

[0087] The computing unit 520 is used to extract features from the input image using a convolutional neural network to obtain a multi-scale feature map of the input image;

[0088] The fusion unit 530 is used to obtain a fused feature map corresponding to each branch structure when performing feature fusion on the multi-scale feature map according to at least two branch structures, and the feature map scale corresponding to each branch structure is different.

[0089] The recognition unit 540 is used to identify the output image corresponding to the input image based on the fusion feature map corresponding to each branch structure, and the output image includes the recognition position of the roadside sign.

[0090] In some embodiments of this application, the roadside sign recognition device 500 further includes a preprocessing unit, which is used to obtain the maximum scale in the multi-scale feature map as the feature map scale corresponding to one of the at least two branch structures, and to set the feature map scale corresponding to the other branch structures in the at least two branch structures to be greater than the maximum scale in the multi-scale feature map.

[0091] In some embodiments of this application, the fusion unit 530 is configured to perform scale transformation on the multi-scale feature maps according to the feature map scale corresponding to each branch structure, to obtain a feature map to be fused with the same scale as the feature map corresponding to each branch structure; to superimpose the feature maps to be fused corresponding to each branch structure to obtain an initial fused feature map corresponding to each branch structure; and to superimpose the initial fused feature map corresponding to the smaller-scale branch structure onto the initial fused feature map corresponding to the larger-scale branch structure according to the feature map scale corresponding to each branch structure, to obtain a fused feature map corresponding to the larger-scale branch structure.

[0092] In some embodiments of this application, the fusion unit 530 is further configured to perform convolution processing on the initial fusion feature map corresponding to the smaller-scale branch structure, and scale transformation on the initial fusion feature map after convolution processing, so that the scale of the scale-transformed initial fusion feature map is the same as the scale of the feature map corresponding to the larger-scale branch structure; and linearly superimpose the scale-transformed initial fusion feature map of the smaller-scale branch structure onto the initial fusion feature map corresponding to the larger-scale branch structure to obtain the fusion feature map corresponding to the larger-scale branch structure.

[0093] In some embodiments of this application, the fusion unit 530 is further configured to compare the scale of each scale feature map with the scale of the feature map corresponding to each branch structure. If the scale of the scale feature map is greater than the scale of the feature map corresponding to the branch structure, the scale feature map is upsampled to obtain a feature map to be fused with the same scale as the feature map corresponding to the branch structure. If the scale of the scale feature map is equal to the scale of the feature map corresponding to the branch structure, the scale feature map is directly used as the feature map to be fused corresponding to the branch structure.

[0094] In some embodiments of this application, the recognition unit 540 is further configured to scale the fusion feature maps corresponding to each branch structure according to the size of the input image, so that the size of each fusion feature map after scale transformation is the same as the size of the input image; and to obtain the output image corresponding to the input image according to the scale-transformed fusion feature map corresponding to each branch.

[0095] In some embodiments of this application, the recognition unit 540 is further configured to linearly superimpose the scale-transformed fused feature maps corresponding to each branch according to the pixel positions to obtain the output image.

[0096] It is understood that the roadside sign recognition device described above can realize all the steps of the roadside sign recognition method provided in the foregoing embodiments. The relevant explanations of the roadside sign recognition method are applicable to the roadside sign recognition device, and will not be repeated here.

[0097] Figure 6 A schematic diagram of an electronic device according to one embodiment of this application is shown. Please refer to... Figure 6 At the hardware level, the electronic device includes a processor and memory, and optionally also includes an internal bus and a network interface. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0098] The processor, interface module, communication module, and memory can be interconnected via an internal bus. This internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0099] Memory is used to store executable instructions for a computer. Memory provides these instructions to the processor via an internal bus.

[0100] The processor executes computer-executable instructions stored in memory and specifically performs the following operations:

[0101] Acquire an input image including roadside signs;

[0102] A convolutional neural network is used to extract features from the input image to obtain a multi-scale feature map of the input image;

[0103] When the multi-scale feature map is fused according to at least two branch structures, a fused feature map corresponding to each branch structure is obtained, and the feature map scale corresponding to each branch structure is different.

[0104] The output image corresponding to the input image is obtained by identifying the fusion feature map corresponding to each branch structure, and the output image includes the identification location of the roadside sign.

[0105] The above is as stated in this application. Figure 1 The functions performed by the road sign corner recognition method disclosed in the illustrated embodiment can be applied to a processor, or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the processor's hardware or through software instructions.

[0106] This application also proposes a computer-readable storage medium that stores one or more programs, which, when executed by a processor, perform the following operations:

[0107] Acquire an input image including roadside signs;

[0108] A convolutional neural network is used to extract features from the input image to obtain a multi-scale feature map of the input image;

[0109] When the multi-scale feature map is fused according to at least two branch structures, a fused feature map corresponding to each branch structure is obtained, and the feature map scale corresponding to each branch structure is different.

[0110] The output image corresponding to the input image is obtained by identifying the fusion feature map corresponding to each branch structure, and the output image includes the identification location of the roadside sign.

[0111] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps that specify the function are in one or more boxes.

[0114] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0115] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0116] Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0117] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0118] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0119] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A roadside sign recognition method characterized by comprising: The method comprises: obtaining an input image including a roadside sign; performing feature extraction on the input image by using a convolutional neural network to obtain a multi-scale feature map of the input image; when performing feature fusion on the multi-scale feature map according to at least two branch structures, obtaining a fused feature map corresponding to each branch structure, and the feature map corresponding to each branch structure has different scales; when performing feature fusion on the multi-scale feature map according to at least two branch structures, obtaining a fused feature map corresponding to each branch structure, comprising: respectively performing scale transformation on the multi-scale feature map according to the scale of the feature map corresponding to each branch structure to obtain a to-be-fused feature map having the same scale as the feature map corresponding to each branch structure; stacking the to-be-fused feature map corresponding to each branch structure to obtain an initial fused feature map corresponding to each branch structure; stacking the initial fused feature map corresponding to a branch structure with a smaller scale to the initial fused feature map corresponding to a branch structure with a larger scale according to the scale of the feature map corresponding to each branch structure to obtain a fused feature map corresponding to the branch structure with the larger scale; recognizing the fused feature map corresponding to each branch structure to obtain an output image corresponding to the input image, and the output image includes a recognized position of the roadside sign.

2. The method of claim 1, wherein, The scales of the feature maps corresponding to the at least two branch structures are set by the following steps: obtaining a maximum scale in the multi-scale feature map as the scale of a feature map corresponding to one of the at least two branch structures, and setting the scales of the feature maps corresponding to the other branch structures in the at least two branch structures to be greater than the maximum scale in the multi-scale feature map.

3. The method of claim 1, wherein, Stacking the initial fused feature map corresponding to a branch structure with a smaller scale to the initial fused feature map corresponding to a branch structure with a larger scale according to the scale of the feature map corresponding to each branch structure to obtain a fused feature map corresponding to the branch structure with the larger scale, comprising: performing convolution processing on the initial fused feature map corresponding to the branch structure with the smaller scale, performing scale transformation on the initial fused feature map after the convolution processing, so that the scale of the initial fused feature map after the scale transformation is the same as the scale of the feature map corresponding to the branch structure with the larger scale; linearly stacking the initial fused feature map after the scale transformation of the branch structure with the smaller scale to the initial fused feature map corresponding to the branch structure with the larger scale to obtain the fused feature map corresponding to the branch structure with the larger scale.

4. The method of claim 1, wherein, Respectively performing scale transformation on the multi-scale feature map according to the scale of the feature map corresponding to each branch structure to obtain a to-be-fused feature map having the same scale as the feature map corresponding to each branch structure, comprising: performing scale comparison between each scale feature map and the scale of the feature map corresponding to each branch structure, if the scale of the scale feature map is greater than the scale of the feature map corresponding to the branch structure, upsampling the scale feature map to obtain a to-be-fused feature map having the same scale as the feature map corresponding to the branch structure; if the scale of the scale feature map is equal to the scale of the feature map corresponding to the branch structure, directly taking the scale feature map as the to-be-fused feature map corresponding to the branch structure.

5. The method of claim 1, wherein, Obtaining an output image corresponding to the input image according to the fused feature map corresponding to each branch, comprising: According to the size of the input image, the scale of each branch structure corresponding fusion feature map is transformed, so that the size of each scale transformed fusion feature map is the same as the size of the input image; According to the scale transformed fusion feature map corresponding to each branch, the output image corresponding to the input image is obtained.

6. The method of claim 5, wherein, According to the scale transformed fusion feature map corresponding to each branch, the output image corresponding to the input image is obtained, including: The scale transformed fusion feature map corresponding to each branch is linearly superimposed according to the pixel position, and the output image is obtained.

7. A roadside sign recognition apparatus characterized by comprising: Including: An acquisition unit is configured to acquire an input image including a roadside sign; A calculation unit is configured to extract features of the input image by using a convolutional neural network to obtain a multi-scale feature map of the input image; A fusion unit is configured to obtain a fusion feature map corresponding to each branch structure when performing feature fusion on the multi-scale feature map according to at least two branch structures, and the feature map corresponding to each branch structure has different scales; When performing feature fusion on the multi-scale feature map according to at least two branch structures, the fusion feature map corresponding to each branch structure is obtained, including: According to the scale of the feature map corresponding to each branch structure, the multi-scale feature map is respectively scaled to obtain a to-be-fused feature map corresponding to each branch structure and having the same scale as the feature map corresponding to each branch structure; The to-be-fused feature map corresponding to each branch structure is superimposed to obtain an initial fusion feature map corresponding to each branch structure; According to the scale of the feature map corresponding to each branch structure, the initial fusion feature map corresponding to a branch structure with a smaller scale is superimposed to the initial fusion feature map corresponding to a branch structure with a larger scale to obtain a fusion feature map corresponding to the branch structure with the larger scale; An identification unit is configured to identify the output image corresponding to the input image according to the fusion feature map corresponding to each branch structure, and the output image includes an identification position of the roadside sign.

8. An electronic device, comprising: Including: A processor; And A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the roadside sign identification method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs, which, when executed by a processor, implement the roadside sign identification method of any one of claims 1-6.

Images