Traffic element recognition method and device

Abstract

The present invention discloses a method and device for identifying traffic elements, and relates to the field of computer technology. A specific implementation of the method includes: obtaining a first feature map of a road image, the first feature map having a first resolution; performing a first conversion process on the first feature map to obtain a feature map in a preset form; performing a convolution operation on the feature map in the preset form to obtain a fused feature map; performing a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution; and performing image recognition processing on the road image using the low-scale feature map to determine the recognition result of the traffic element corresponding to the road image. This implementation can improve the accuracy of identifying traffic elements in autonomous driving technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for identifying traffic elements. Background Art

[0002] With the development of computer vision technology, especially image processing technology, target-based image acquisition and recognition technology has emerged, making autonomous driving possible. At present, the recognition technology for traffic elements in autonomous driving has shifted from traditional feature extraction methods to deep learning methods, but the existing recognition technology for traffic elements often has low accuracy, which brings safety risks to autonomous driving. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for identifying traffic elements, which can improve the accuracy of identifying traffic elements in autonomous driving technology.

[0004] In a first aspect, an embodiment of the present invention provides a method for identifying traffic elements, including:

[0005] Acquire a first feature map of a road image, wherein the first feature map has a first resolution;

[0006] Performing a first conversion process on the first feature map to obtain a feature map in a preset form;

[0007] Performing a convolution operation on the feature map in the preset form to obtain a fused feature map;

[0008] Performing a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution;

[0009] The low-scale feature map is used to perform image recognition processing on the road image to determine a recognition result of a traffic element corresponding to the road image.

[0010] Optionally, before performing image recognition processing on the road image using the low-scale feature map, the method further includes:

[0011] Acquire a second feature map of the road image, where a resolution of the second feature map is the second resolution;

[0012] The step of performing image recognition processing on the road image by using the low-scale feature map includes:

[0013] Generate a multi-scale feature map according to the low-scale feature map and the second feature map;

[0014] The multi-scale feature map is used to perform image recognition processing on the road image.

[0015] Optionally, generating a multi-scale feature map according to the low-scale feature map and the second feature map includes:

[0016] Using at least one dilated convolution, respectively perform a convolution operation on the second feature map to obtain at least one extended feature map, wherein the resolution of each of the extended feature maps is greater than the second resolution;

[0017] Generating a combined feature map according to the at least one extended feature map;

[0018] Performing a third conversion process on the stitched feature map to obtain a high-scale feature map, where the resolution of the high-scale feature map is the second resolution;

[0019] The low-scale feature map and the high-scale feature map are combined to generate the multi-scale feature map.

[0020] Optionally, generating a stitched feature map according to the at least one extended feature map includes:

[0021] Determining a maximum extended feature map having a maximum resolution from the at least one extended feature map, wherein the resolution of the maximum extended feature map is a third resolution;

[0022] Amplifying other extended feature maps in the at least one extended feature map so that the resolution of the other extended feature maps after the processing is the third resolution;

[0023] The other processed extended feature maps and the maximum extended feature map are combined to generate the combined feature map.

[0024] Optionally, generating a multi-scale feature map according to the low-scale feature map and the second feature map includes:

[0025] The low-scale feature map and the second feature map are combined to generate the multi-scale feature map.

[0026] Optionally, performing image recognition processing on the road image using the multi-scale feature map includes:

[0027] The multi-scale feature map is detected by using a detection head to perform image recognition processing on the road image.

[0028] Optionally, performing a first conversion process on the first feature map to obtain a feature map in a preset form includes:

[0029] Performing pooling processing on the first feature map to obtain a pooled feature map;

[0030] Determine position information of non-retained points corresponding to the pooled feature map in the first feature map;

[0031] Moving the data of the non-reserved points in the first feature map to the back of the data of the reserved points in the pooled feature map to generate a reserved feature map;

[0032] The retained feature map is spliced ​​by channel to obtain a feature map in the form of 1*1*n.

[0033] In a second aspect, an embodiment of the present invention provides a traffic element recognition device, comprising:

[0034] A map acquisition module, configured to acquire a first feature map of a road image, wherein the first feature map has a first resolution;

[0035] A first conversion module, used for performing a first conversion process on the first feature map to obtain a feature map in a preset form;

[0036] A convolution operation module, used for performing a convolution operation on the feature map in the preset form to obtain a fused feature map;

[0037] A second conversion module, configured to perform a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution;

[0038] The image recognition module is used to perform image recognition processing on the road image using the low-scale feature map to determine the recognition result of the traffic element corresponding to the road image.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0040] one or more processors;

[0041] a storage device for storing one or more programs,

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the above embodiments.

[0044] An embodiment of the above invention has the following advantages or beneficial effects: applying the method of the embodiment of the present invention, a low-scale feature map is generated according to the first feature map. The low-scale feature map retains the information in the previous first feature map, and the resolution of the low-scale feature map is smaller than the resolution of the first feature map, that is, the low-scale feature map retains the information in the larger resolution feature map through a smaller resolution. Using the low-scale feature map to perform image recognition processing on road images can improve the detection capability of small targets, improve the accuracy of identifying traffic elements in autonomous driving technology, and thus improve the safety of the autonomous driving process.

[0045] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0047] Figure 1 is a schematic diagram of a process of a method for identifying traffic elements provided by an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of the process of another method for identifying traffic elements provided by an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of a process of another method for identifying traffic elements provided by an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a network structure based on image detection provided by an embodiment of the present invention;

[0051] Figure 5 It is a structural schematic diagram of a traffic element recognition device provided by an embodiment of the present invention;

[0052] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0054] To ensure the safety of autonomous driving, autonomous vehicles must detect and identify all obstacles within their field of view. Object detection is a key algorithm to achieve this goal. Since deep learning, detection algorithms have evolved from single-scale detection to multi-scale detection. Increasing the ability of algorithms to detect at different scales is an important way to improve the performance of detection algorithms. At present, the core idea of ​​existing multi-scale technologies such as SSD and FPN is to construct feature maps of different scales through different downsampling ratios, so that feature maps of different scales have different sizes of receptive fields, thereby achieving the purpose of multi-scale detection. Since feature maps of multiple scales need to be calculated in this process, and each feature map of a scale requires an independent detection head for detection, this structure undoubtedly increases the amount of calculation. In the autonomous driving environment, due to the limitation of computing power, such structures are usually not used, or only simplified structures can be used, such as reducing the number of multi-scale feature maps, resulting in a decrease in the algorithm's ability to detect at multiple scales.

[0055] Mainstream detection algorithms can usually be divided into one-stage networks, such as SSD, YOLO, RetinaNet, etc., or two-stage networks, such as the RCNN series. In fact, whether it is a one-stage or two-stage algorithm, the idea of ​​multi-scale detection is reflected in its algorithm: for example, the classic two-stage network Faster RCNN generates Region proposals of various scales through RPN (Region proposal Network), and then unifies the scale through ROI Pooling. The one-stage network usually has a visible multi-scale structure, such as SSD intercepts the feature maps at different downloading magnifications of the backbone network (backbone), and then performs detection on these feature maps; FPN adds a top-down path based on SSD, so that feature maps of different scales can be fused, and then detection is performed on feature maps of different scales. And structures like FPN or PANet can even be applied to the RPN subnet in the two-stage network to increase the detection ability of foreground targets.

[0056] The main problem with applying this structure usually comes from the challenge of computing power, especially in the environment of autonomous driving. Recently, the academic community has also been exploring whether it is possible to complete multi-scale detection using only one scale of feature map, such as YOLOF. It uses a dilated encoder to perform multi-scale encoding on a feature map of a special scale in the backbone network, such as the C5 layer in ResNet50 (downsampling ratio 32), but the biggest challenge it faces is that it can only produce feature maps larger than the C5 scale. If a target is smaller than the scale of the C5 feature map, then its effect will be greatly affected. The experiments in the YOLOF paper also show that its detection ability for small targets is much inferior to that of large targets. In addition, the structure of the transformer model can also simplify this multi-scale network structure, but its training time is 7 to 10 times that of the mainstream algorithm, and it is not suitable for scenarios with faster iterations such as autonomous driving.

[0057] Based on this, an embodiment of the present invention designs a method for identifying traffic elements, which can be applied to visual target detection in autonomous driving, so that the generated feature map retains the information in the previous feature map with a larger resolution, improves the detection capability of small targets, and makes the detection results more accurate.

[0058] Figure 1 FIG. 1 is a schematic diagram of a process of a method for identifying traffic elements provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0059] Step 101: Acquire a first feature map of a road image, where the first feature map has a first resolution.

[0060] The road image can be input into a preset neural network, and feature maps at different downsampling magnifications of the preset neural network can be intercepted as the first feature map. The preset neural network can be a convolutional neural network, a backbone network, etc. The preset neural network can use lenet-5, ResNet-34, ResNet-50, etc. The first feature map can use C1 and C3 in lenet-5, and can also use C3 and C4 in ResNet-50, etc.

[0061] Step 102: Perform a first conversion process on the first feature map to obtain a feature map in a preset form.

[0062] The data contained in the first feature map after conversion is the same as the data contained in the first feature map before conversion, that is, the information in the first feature map after the first conversion process is not lost. The preset form can be set according to specific needs. For example, if the first feature map before conversion is in the form of 2*2*2, the first feature map after conversion can be in the form of 1*1*8. If the first feature map before conversion is in the form of 4*4*2, the first feature map after conversion can be in the form of 1*1*32, and can also be converted to the form of 2*2*8, etc.

[0063] As an implementable method, the preset form is 1*1*n. Converting the first feature map into 1*1*n form can make the information in the feature map better integrated in the convolution operation of the subsequent step 103, and it is easier to select which positions or combinations of positions in the feature map may represent the information of low-scale targets.

[0064] The first feature map is converted into a 1*1*n format, and the information in the first feature map is not lost, and the subsequent processing of the feature map is convenient. The data of each position in the first feature map can be taken out in sequence according to a preset order, and the taken out data is put into the corresponding position to convert the first feature map into a feature map in a preset format of 1*1*n, where n is a positive integer.

[0065] The first feature map can also be subjected to a first conversion process by the following method, including: performing pooling processing on the first feature map to obtain a pooling feature map; determining position information of non-retained points corresponding to the pooling feature map in the first feature map; moving the data of the non-retained points in the first feature map to behind the data of the retained points of the pooling feature map to generate a retained feature map; and performing channel-wise splicing processing on the retained feature map.

[0066] Perform maximum pooling on the first feature map to obtain a pooled feature map. The convolution kernel size and convolution step size used in the pooling process can be set according to the structure of the first feature map. Then add the data of the non-reserved points in the first feature map to the pooled feature map to generate a retained feature map. Finally, the retained feature maps are spliced ​​together by channel to obtain a feature map in the form of 1*1*n.

[0067] Step 103: Perform a convolution operation on the feature map in a preset form to obtain a fused feature map. Performing a convolution operation on the feature map in a preset form can fuse the information in the feature map in the preset form to retain more information in the feature map. The convolution kernel can be selected according to specific needs and the form of the first feature map after conversion. For example: a 1*1 convolution kernel, a 3*3 convolution kernel. The fused feature map obtained after the convolution operation retains the information in the first feature map with a resolution of the first resolution.

[0068] Step 104: Perform a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution, where the second resolution is smaller than the first resolution.

[0069] The fused feature map may be processed by dimension increase or dimension reduction using a 1*1 convolution kernel to generate a low-scale feature map. Through the processing from step 101 to step 104, although the low-scale feature map has a smaller resolution, the low-scale feature map retains information in the first feature map having a first resolution.

[0070] Step 105: Using the low-scale feature map, perform image recognition processing on the road image to determine the recognition result of the traffic element corresponding to the road image.

[0071] Traffic elements are objects that can affect driving control in autonomous driving. Traffic elements may include: vehicles, pedestrians, traffic lights, lane lines, road signs, etc. The recognition results of traffic elements include: the location, size, direction, category and other information of each traffic element.

[0072] In an embodiment of the present invention, a low-scale feature map is generated based on the first feature map. The low-scale feature map retains the information in the previous first feature map, and the resolution of the low-scale feature map is smaller than the resolution of the first feature map, that is, the low-scale feature map retains the information in the larger resolution feature map through a smaller resolution. Using the low-scale feature map to perform image recognition processing on the road image can improve the detection capability of small targets and improve the accuracy of identifying traffic elements in the autonomous driving technology.

[0073] Figure 2 FIG. 1 is a schematic diagram of the process of another method for identifying traffic elements provided by an embodiment of the present invention. Figure 2 As shown, the method includes:

[0074] Step 201: Acquire a first feature map of a road image, where the first feature map has a first resolution.

[0075] Step 202: Perform a first conversion process on the first feature map to obtain a feature map in the form of 1*1*n.

[0076] Step 203: Perform a convolution operation on the feature map in the form of 1*1*n with a 1*1 convolution kernel to obtain a fused feature map.

[0077] Step 204: performing a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution, where the second resolution is smaller than the first resolution.

[0078] Step 205: Acquire a second feature map of the road image, where the resolution of the second feature map is a second resolution.

[0079] Step 206: Generate a multi-scale feature map based on the low-scale feature map and the second feature map.

[0080] Since the low-scale feature map and the second feature map have the same resolution, the low-scale feature map and the second feature map can be directly spliced ​​to generate a multi-scale feature map. The spliced ​​low-scale feature map and the second feature map can also be subjected to convolution, pooling and other processing to generate a multi-scale feature map.

[0081] Step 207: Using the multi-scale feature map, perform image recognition processing on the road image.

[0082] In the embodiment of the present invention, the multi-scale feature map retains the information in the feature maps of multiple scales at the same scale, and the multi-scale detection can be completed only on the feature map of a single scale, which can greatly reduce the computational complexity of the multi-scale detection of the network compared to the mainstream solution. Since the current target detection of autonomous driving is mostly calculated on the vehicle side, and the calculation examples on the vehicle side are usually limited, compared with the mainstream solution, the method of the embodiment of the present invention has significantly lower demand for computing power and is suitable for the scenario of autonomous driving.

[0083] As a specific embodiment, a detection head can be used to detect multi-scale feature maps to perform image recognition processing on road images. The detection head includes a classification method and a BBox regression (Bounding Box Regression) method for each traffic element. In the prior art, for feature maps of multiple scales, each feature map of a scale requires an independent detection head for detection. The multi-scale feature map in the embodiment of the present invention integrates the information in feature maps of multiple scales at the same scale, and the detection can be completed through the same detection head, reducing the amount of calculation and time required to identify traffic elements.

[0084] Figure 3 FIG. 1 is a schematic diagram of a flow chart of another method for identifying traffic elements provided by an embodiment of the present invention. Figure 3 As shown, the method includes:

[0085] Step 301: Acquire a first feature map of a road image, where the first feature map has a first resolution.

[0086] Step 302: Perform a first conversion process on the first feature map to obtain a feature map in the form of 1*1*n.

[0087] Step 303: Perform a convolution operation on the feature map in the form of 1*1*n with a 1*1 convolution kernel to obtain a fused feature map.

[0088] Step 304: Perform a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution, where the second resolution is smaller than the first resolution.

[0089] Step 305: Acquire a second feature map of the road image, where the resolution of the second feature map is a second resolution.

[0090] Step 306: Use at least one dilated convolution to perform convolution operations on the second feature map to obtain at least one extended feature map, and the resolution of each extended feature map is greater than the second resolution.

[0091] The dilation ratio of the dilated convolution can be set according to the needs. The dilation ratio can be: 2, 4, 8, etc.

[0092] Step 307: Generate a stitched feature map based on at least one extended feature map.

[0093] The stitched feature map can be generated by the following method: from the at least one extended feature map, a maximum extended feature map with a maximum resolution is determined, and the resolution of the maximum extended feature map is a third resolution; other extended feature maps in the at least one extended feature map are amplified so that the resolution of the other extended feature maps after processing is the third resolution; and the other extended feature maps after processing and the maximum extended feature map are stitched together to generate the stitched feature map.

[0094] Step 308: performing a third conversion process on the stitched feature map to obtain a high-scale feature map, where the resolution of the high-scale feature map is the second resolution.

[0095] The dimensionality of the combined feature map is increased or decreased to generate a high-scale feature map. Through the processing from step 305 to step 308, information in multiple feature maps with a resolution greater than the second resolution is retained in the high-scale feature map.

[0096] Step 309: combine the low-scale feature map and the high-scale feature map to generate a multi-scale feature map.

[0097] Since the low-scale feature map and the high-scale feature map have the same resolution, the low-scale feature map and the high-scale feature map can be directly spliced ​​to generate a multi-scale feature map. The feature map obtained by splicing the low-scale feature map and the high-scale feature map can also be subjected to convolution, pooling and other processing to generate a multi-scale feature map.

[0098] By performing convolution operations on the second feature map with different dilation ratios, different scale receptive fields are introduced into the generated high-scale feature map, and then a multi-scale feature map is generated based on the low-scale feature map and the high-scale feature map. Compared with the method of directly combining the low-scale feature map and the second feature map to generate a multi-scale feature map, the detection result can be more accurate due to the additional introduction of different scale receptive fields.

[0099] Step 310: Perform image recognition processing on the road image using the multi-scale feature map.

[0100] In the embodiment of the present invention, the multi-scale feature map retains information in the feature map with a resolution greater than the second resolution, and also retains information in the feature map with a resolution less than the second resolution. By using the multi-scale feature map for image recognition processing, multi-scale detection can be completed on a single-scale feature map, which can greatly reduce the amount of calculation required for multi-scale detection and has a high accuracy detection effect, which is particularly suitable for autonomous driving scenarios.

[0101] In an embodiment of the present invention, an efficient multi-scale traffic element detection method is designed. The method can generate a feature map containing multi-scale information. The traffic element detection method can complete multi-scale detection only on this feature map. In order to achieve multi-scale detection on a feature map, the main challenge is how to represent features of multiple scales on a feature map. This problem is further decomposed into two sub-problems: 1. After intercepting a feature map of a specific scale from the backbone network, how to retain the information in the feature map of a larger resolution (i.e., a smaller scale). 2. How to represent information larger than its scale on this feature map. In comparison, the first sub-problem is more challenging because, usually, the shallower feature maps in the backbone network (the part containing low-scale information) have been downsampled through convolution or pooling operations, and low-scale information has been lost; the second problem does not have the problem of information loss, similar to YOLOF, and dilated convolution can be used to extract larger-scale information.

[0102] Based on the above analysis, an embodiment of the present invention proposes a network based on image detection. Figure 4 FIG. 1 is a schematic diagram of a network structure based on image detection provided by an embodiment of the present invention. Figure 4As shown, an embodiment of the present invention designs a multi-scale feature extractor (MSFE) to solve the above two problems. The input of the multi-scale feature extractor is two feature maps of different scales of the backbone network, such as C4 (downsampling ratio 16) and C5 (downsampling ratio 32) in ResNet-50. MSFE includes two sub-modules, a low-resolution preservation module (LRP, LowResolution Preservation) for preserving low-scale information and generating a feature map with the same resolution as C5, and an extended encoder (DE, Dilated Encoder) for extracting larger scale information. In this way, MSFE can use the scale of C5 as a benchmark to generate a feature map P4 that is larger than C5 by P5 or smaller than C5, and finally obtain a feature map (Multi-scaled Feature-map) containing multiple scale information. Only one detection head is required, thereby reducing the computational complexity. The detailed structures of the two modules LRP and DE are as follows:

[0103] Low Resolution Preservation (LRP), this submodule is used to preserve information smaller than the scale of C5. Since the information of the lower layer comes from the feature map with higher resolution, such as C4, that is, its resolution is not the same as that of C5, this case designs a maximum spatial compression operation (Max Spatial Squeeze) to unify the resolution of C4 and C5, so as to finally form a multi-scale feature map (Multi-scaled Feature-map). Given an input C4, the specific implementation steps of maximum spatial compression are: 1) Perform a maximum pooling (max-pooling, kernel-size=2, stride=2) operation on C4, and obtain the position information of the retained value in C4 during the pooling operation. 2) Move the position of the non-maximum value in the convolution kernel to the back of the maximum value, and concatenate them by channel to obtain S4, as shown Figure 4 3) A 1X1 convolution is used as the channel attention, the purpose is to select which positions or combinations of positions in the convolution kernel may represent the information S4' of the low-scale target, and finally output the feature map P4, the resolution of P4 is consistent with C5.

[0104] Dilated Encoder (DE): This submodule is used to extract information of a larger scale than C5. Specifically, it uses dilated convolutions with different dilation ratios to extract information of different scales, and then stitches them together to output the feature map P5. The resolution of P5 is consistent with that of C5.

[0105] To make the method of the embodiment of the present invention easier to understand. Figure 4 The network structure shown, an embodiment of the present invention proposes an efficient multi-scale traffic element detection method, the method comprising:

[0106] S1: Network training. The training data is the RGB camera images collected by the autonomous driving vehicle and the artificial target frame annotation data. Except for the following special instructions, the present embodiment of the present invention can adopt the current mainstream detector training process.

[0107] Select a backbone network of a general feature extractor (the solution of the embodiment of the present invention takes ResNet-50 as an example, and other backbone networks can actually be used instead), load the ImageNet pre-trained model of ResNet-50, unlock all layers, and initialize the feature extractor.

[0108] The loss function uses focal loss (as shown in Formula 1), where y is the ground-truth category. In the early stage of training, α=0.5 and γ=1 are set. In the later stage, as the number of samples in the sample library increases, α=0.25 and γ=2 are set.

[0109] L=-α t (1-P t ) γ log(P t )

[0110]

[0111] BBOX regression uses Smooth-L1 as the loss function.

[0112] S2: Deploy the model in the autonomous driving scenario. This step includes the following sub-steps:

[0113] S21: Publish the image data collected by the RGB cameras in each direction to the system bus.

[0114] S22: Get RGB camera data from the bus.

[0115] S23: Preprocess the RGB image, including adjusting the image size, removing image distortion, and normalizing the image.

[0116] S24: Input the preprocessed data into the model to obtain the prediction result of the target, including information of position, size, orientation, and category.

[0117] S25: Publish the prediction result to the system bus for use by downstream modules. The method of the embodiment of the present invention shortens the time of model prediction by designing a new network structure.

[0118] In an embodiment of the present invention, an efficient multi-scale traffic element detection method is designed, which can be applied to visual target detection in autonomous driving. Its characteristic is that it can generate a feature map containing multi-scale information, and the algorithm can complete multi-scale detection only on this feature map. Compared with mainstream solutions such as SSD or FPN, it greatly reduces the computational complexity of multi-scale detection of the network. Since the current target detection of autonomous driving is mostly calculated on the vehicle side, and the calculation examples on the vehicle side are usually limited, compared with the mainstream solution, it is suitable for the scenario of autonomous driving because its demand for computing power is significantly reduced.

[0119] Figure 5 FIG. 1 is a schematic diagram of a traffic element recognition device provided by an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0120] A map acquisition module 501 is used to acquire a first feature map of a road image, wherein the first feature map has a first resolution;

[0121] A first conversion module 502, configured to perform a first conversion process on the first feature map to obtain a feature map in a preset form;

[0122] A convolution operation module 503 is used to perform a convolution operation on the feature map in the preset form to obtain a fused feature map;

[0123] A second conversion module 504 is used to perform a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution;

[0124] The image recognition module 505 is used to perform image recognition processing on the road image using the low-scale feature map to determine the recognition result of the traffic element corresponding to the road image.

[0125] Optionally, the image acquisition module 501 is further used to acquire a second feature image of the road image, where the resolution of the second feature image is the second resolution;

[0126] An image recognition module 505 is specifically configured to generate a multi-scale feature map according to the low-scale feature map and the second feature map;

[0127] The multi-scale feature map is used to perform image recognition processing on the road image.

[0128] Optionally, the image recognition module 505 is specifically configured to use at least one dilated convolution to perform a convolution operation on each of the second feature maps to obtain at least one extended feature map, wherein a resolution of each of the extended feature maps is greater than the second resolution;

[0129] Generating a combined feature map according to the at least one extended feature map;

[0130] Performing a third conversion process on the stitched feature map to obtain a high-scale feature map, where the resolution of the high-scale feature map is the second resolution;

[0131] The low-scale feature map and the high-scale feature map are combined to generate the multi-scale feature map.

[0132] Optionally, the image recognition module 505 is specifically configured to determine a maximum extended feature map having a maximum resolution from the at least one extended feature map, wherein the resolution of the maximum extended feature map is a third resolution;

[0133] Amplifying other extended feature maps in the at least one extended feature map so that the resolution of the other extended feature maps after the processing is the third resolution;

[0134] The other processed extended feature maps and the maximum extended feature map are combined to generate the combined feature map.

[0135] Optionally, the image recognition module 505 is specifically configured to combine the low-scale feature map and the second feature map to generate the multi-scale feature map.

[0136] Optionally, the image recognition module 505 is specifically configured to use a detection head to detect the multi-scale feature map so as to perform image recognition processing on the road image.

[0137] Optionally, the first conversion module 502 is specifically configured to perform pooling processing on the first feature map to obtain a pooled feature map;

[0138] Determine position information of non-retained points corresponding to the pooled feature map in the first feature map;

[0139] Moving the data of the non-reserved points in the first feature map to the back of the data of the reserved points in the pooled feature map to generate a reserved feature map;

[0140] The retained feature map is spliced ​​by channel to obtain a feature map in the form of 1*1*n.

[0141] An embodiment of the present invention provides an electronic device, including:

[0142] one or more processors;

[0143] a storage device for storing one or more programs,

[0144] When one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0145] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present invention. Figure 6 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0146] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0148] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0149] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0150] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0151] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: a graph acquisition module, a first conversion module, a convolution operation module, a second conversion module, and an image recognition module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the graph acquisition module may also be described as a "module for acquiring a first feature map of a road image."

[0152] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0153] Acquire a first feature map of a road image, wherein the first feature map has a first resolution;

[0154] Performing a first conversion process on the first feature map to obtain a feature map in a preset form;

[0155] Performing a convolution operation on the feature map in the preset form to obtain a fused feature map;

[0156] Performing a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution;

[0157] The low-scale feature map is used to perform image recognition processing on the road image to determine a recognition result of a traffic element corresponding to the road image.

[0158] According to the technical solution of the embodiment of the present invention, the method of the embodiment of the present invention is applied to generate a low-scale feature map according to the first feature map. The low-scale feature map retains the information in the previous first feature map, and the resolution of the low-scale feature map is smaller than the resolution of the first feature map. Using the low-scale feature map to perform image recognition processing on the road image can improve the detection capability of small targets and improve the accuracy of identifying traffic elements in autonomous driving technology.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying traffic elements, characterized in that: include: Acquire a first feature map of a road image, wherein the first feature map has a first resolution; Performing a first conversion process on the first feature map to obtain a feature map in a preset form; wherein information in the first feature map after the first conversion process is not lost; Performing a convolution operation on the feature map in the preset form to obtain a fused feature map; Performing a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution; The low-scale feature map is used to perform image recognition processing on the road image to determine a recognition result of a traffic element corresponding to the road image.

2. The method according to claim 1, characterized in that Before performing image recognition processing on the road image using the low-scale feature map, the method further includes: Acquire a second feature map of the road image, where a resolution of the second feature map is the second resolution; The step of performing image recognition processing on the road image by using the low-scale feature map includes: Generate a multi-scale feature map according to the low-scale feature map and the second feature map; The multi-scale feature map is used to perform image recognition processing on the road image.

3. The method according to claim 2, characterized in that Generating a multi-scale feature map according to the low-scale feature map and the second feature map includes: Using at least one dilated convolution, respectively perform a convolution operation on the second feature map to obtain at least one extended feature map, wherein the resolution of each of the extended feature maps is greater than the second resolution; Generating a combined feature map according to the at least one extended feature map; Performing a third conversion process on the stitched feature map to obtain a high-scale feature map, where the resolution of the high-scale feature map is the second resolution; The low-scale feature map and the high-scale feature map are combined to generate the multi-scale feature map.

4. The method according to claim 3, characterized in that The step of generating a stitched feature map according to the at least one extended feature map comprises: Determining a maximum extended feature map having a maximum resolution from the at least one extended feature map, wherein the resolution of the maximum extended feature map is a third resolution; Amplifying other extended feature maps in the at least one extended feature map so that the resolution of the other extended feature maps after the processing is the third resolution; The other processed extended feature maps and the maximum extended feature map are combined to generate the combined feature map.

5. The method according to claim 2, characterized in that: Generating a multi-scale feature map according to the low-scale feature map and the second feature map includes: The low-scale feature map and the second feature map are combined to generate the multi-scale feature map.

6. The method according to claim 2, characterized in that The method of performing image recognition processing on the road image by using the multi-scale feature map includes: The multi-scale feature map is detected by using a detection head to perform image recognition processing on the road image.

7. The method according to claim 1, characterized in that The performing a first conversion process on the first feature map to obtain a feature map in a preset form includes: Performing pooling processing on the first feature map to obtain a pooled feature map; Determine position information of non-retained points corresponding to the pooled feature map in the first feature map; According to the position information, move the data of the non-reserved points in the first feature map to behind the data of the reserved points in the pooled feature map to generate a reserved feature map; The retained feature map is spliced ​​by channel to obtain a feature map in the form of 1*1*n.

8. A traffic element recognition device, characterized in that: include: A map acquisition module, configured to acquire a first feature map of a road image, wherein the first feature map has a first resolution; A first conversion module, used to perform a first conversion process on the first feature map to obtain a feature map in a preset form; wherein the information in the first feature map after the first conversion process is not lost; A convolution operation module, used for performing a convolution operation on the feature map in the preset form to obtain a fused feature map; A second conversion module, configured to perform a second conversion process on the fused feature map to obtain a low-scale feature map with a second resolution; wherein the second resolution is smaller than the first resolution; The image recognition module is used to perform image recognition processing on the road image using the low-scale feature map to determine the recognition result of the traffic element corresponding to the road image.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images