Target detection model training method and device, target detection method and device
By adopting the enhanced convertible atrous convolutional network in monocular 3D target detection, extracting multi-scale features and performing feature fusion, the shortcomings of multi-scale target detection in complex scenes are solved and higher detection accuracy is achieved.
Patent Information
- Application Number
- CN202210435047.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-24
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2042-04-24
AI Technical Summary
Existing monocular 3D object detection has difficulty in effectively handling multi-scale object detection in complex road scenes, especially because the multi-scale problem is not fully considered in deep networks, resulting in insufficient detection capabilities.
An enhanced convertible atrous convolutional network is used to extract multi-scale feature information through the backbone network, and a feature pyramid network is used for feature fusion. The detection head network is combined for target detection, and the loss value is used to update the model parameters to improve detection accuracy.
The accuracy of target detection is improved. Experiments show that the mAP is increased by 1.2 to 1.4 points, and the detection ability of multi-scale targets is enhanced.
Smart Images
Figure CN114694005B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target detection technology, and in particular to a target detection model training method and device, and a target detection method and device. Background Art
[0002] Monocular 3D object detection requires only a single RGB image as input. In complex road scenes, the inherent scales of different detection targets, such as vehicles and pedestrians, vary significantly. The same target, depending on its distance from the observation device, appears at different scales on the RGB image.
[0003] Existing monocular 3D object detection processes input an RGB image into a monocular 3D object detection network consisting of a backbone network, a feature pyramid network, and a detection head. This network outputs a vector containing the target category, target 3D box encoding, target orientation category, target attributes, and target center. The 3D box decoding post-processing module decodes the target 3D box encoding vector to obtain decoded information such as the target's position, scale, depth, and angle. This information is then combined with the score and the NMS (Non-Maximum Suppression) algorithm to determine the final 3D object detection result.
[0004] However, the above-mentioned solutions focus on processing the target scale problem in deep networks. The feature pyramid network first outputs multi-scale feature maps, and then trains the feature maps at different scale levels directly based on the annotation information of targets of different scales. Shallow networks mostly use the general ResNet50 or RestNet101 as the backbone network, and do not consider the multi-scale problem in the extraction of underlying features, resulting in insufficient detection capabilities for multi-scale targets. Summary of the Invention
[0005] The embodiments of the present application provide a target detection model training method and device, and a target detection method and device to extract multi-scale feature information of the target and improve the accuracy of target detection.
[0006] The embodiments of this application adopt the following technical solutions:
[0007] In a first aspect, an embodiment of the present application provides a method for training a target detection model, wherein the method for training a target detection model includes:
[0008] Acquire an image to be trained, wherein the image to be trained includes annotation information of the image to be trained;
[0009] Performing feature extraction on the image to be trained using a backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained;
[0010] Performing feature fusion on the multi-scale feature map of the image to be trained using the feature pyramid network of the target detection model to obtain a fused multi-scale feature map;
[0011] Detecting the fused multi-scale feature map using the detection head network of the target detection model to obtain a target detection result of the image to be trained;
[0012] A loss value is determined according to the target detection result of the image to be trained and the annotation information of the image to be trained, and the parameters of the target detection model are updated using the loss value to obtain a trained target detection model.
[0013] Optionally, the backbone network further includes a first global context module and a second global context module, the enhanced convertible dilated convolution includes a plurality of stages of convertible dilated convolution cascaded in sequence, and the plurality of stages of convertible dilated convolution include at least a first stage convertible dilated convolution and a second stage convertible dilated convolution,
[0014] The step of extracting features from the image to be trained using the backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained includes:
[0015] Obtaining a first feature map of the image to be trained, where the first feature map is output by an upstream module corresponding to the first global context module;
[0016] Processing the first feature map using the first global context module to obtain the second feature map;
[0017] Processing the second feature map using a plurality of stages of convertible dilated convolutions cascaded sequentially to obtain a third feature map;
[0018] The third feature map is processed using the second global context module to obtain a multi-scale feature map of the image to be trained.
[0019] Optionally, the processing the first feature map using the first global context module to obtain the second feature map includes:
[0020] Performing global average pooling on the first feature map to obtain a first global average pooling result;
[0021] Performing a 1x1 convolution process on the first global average pooling process result to obtain a first 1x1 convolution process result;
[0022] The first feature map and the first 1x1 convolution processing result are fused to obtain the second feature map.
[0023] Optionally, the convertible dilated convolution of the first stage includes a first conversion function, a convertible dilated convolution corresponding to a first dilation rate, and a convertible dilated convolution corresponding to a second dilation rate, and the processing of the second feature map by using the convertible dilated convolution of multiple stages cascaded in sequence to obtain the third feature map includes:
[0024] Processing the second feature map using the first conversion function to obtain a processing result of the first conversion function;
[0025] Performing a 3x3 dilated convolution on the second feature map using the convertible dilated convolution corresponding to the first dilated rate to obtain a first dilated convolution result;
[0026] Performing a 3x3 dilated convolution on the second feature map using a convertible dilated convolution corresponding to the second dilated rate to obtain a second dilated convolution result;
[0027] According to the processing result of the first conversion function, the first void convolution processing result and the second void convolution processing result are fused to obtain the fourth feature map of the convertible void convolution output of the first stage.
[0028] Optionally, the first conversion function includes 5x5 average pooling and 1x1 convolution, and the processing of the second feature map by using the first conversion function to obtain a processing result of the first conversion function includes:
[0029] Perform 5x5 average pooling on the second feature map to obtain a first average pooling result;
[0030] Perform a 1x1 convolution on the first average pooling result to obtain a processing result of the first conversion function.
[0031] Optionally, the convertible dilated convolution of the second stage includes a convertible dilated convolution corresponding to a second conversion function and a third dilated rate, and the convertible dilated convolution of the second feature map is processed by sequentially cascading multiple stages to obtain a third feature map, including:
[0032] Processing the second feature map using the second conversion function to obtain a processing result of the second conversion function;
[0033] Performing a 3x3 dilated convolution on the second feature map using the convertible dilated convolution corresponding to the third dilated rate to obtain a third dilated convolution result;
[0034] Obtain a fourth feature map output by the convertible dilated convolution in the first stage;
[0035] According to the processing result of the second conversion function, the fourth feature map and the third hole convolution processing result are fused to obtain the third feature map.
[0036] Optionally, the second conversion function includes 11x11 average pooling and 1x1 convolution, and the processing of the second feature map using the second conversion function to obtain a processing result of the second conversion function includes:
[0037] Perform 11x11 average pooling on the second feature map to obtain a second average pooling result;
[0038] Perform a 1x1 convolution on the second average pooling result to obtain a processing result of the second conversion function.
[0039] In a second aspect, an embodiment of the present application further provides a target detection method, wherein the target detection method includes:
[0040] Obtain the image to be detected;
[0041] Detecting the image to be detected using a target detection model to obtain a target detection result;
[0042] The target detection model is trained based on any of the aforementioned target detection model training methods.
[0043] In a third aspect, an embodiment of the present application further provides a target detection model training device, wherein the target detection model training device includes:
[0044] A first acquisition unit is configured to acquire an image to be trained, wherein the image to be trained includes annotation information of the image to be trained;
[0045] a feature extraction unit, configured to perform feature extraction on the image to be trained using a backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained;
[0046] a feature fusion unit, configured to perform feature fusion on the multi-scale feature map of the image to be trained using the feature pyramid network of the target detection model to obtain a fused multi-scale feature map;
[0047] A first detection unit is configured to detect the fused multi-scale feature map using a detection head network of the target detection model to obtain a target detection result of the image to be trained;
[0048] An updating unit is used to determine a loss value based on the target detection result of the image to be trained and the annotation information of the image to be trained, and to use the loss value to update the parameters of the target detection model to obtain a trained target detection model.
[0049] In a fourth aspect, an embodiment of the present application further provides a target detection device, wherein the target detection device includes:
[0050] A second acquisition unit, configured to acquire an image to be detected;
[0051] A second detection unit is used to detect the image to be detected using a target detection model to obtain a target detection result;
[0052] The target detection model is trained based on the aforementioned target detection model training device.
[0053] In a fifth aspect, an embodiment of the present application further provides an electronic device, including:
[0054] processor; and
[0055] A memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned target detection model training methods or target detection methods.
[0056] In the sixth aspect, an embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes any of the aforementioned target detection model training methods or target detection methods.
[0057] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the target detection model training method of the embodiment of the present application first obtains a to-be-trained image, which includes the annotation information of the to-be-trained image; then uses the backbone network of the target detection model to extract features of the to-be-trained image to obtain a multi-scale feature map of the to-be-trained image, the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the to-be-trained image; then uses the feature pyramid network of the target detection model to perform feature fusion on the multi-scale feature map of the to-be-trained image to obtain a fused multi-scale feature map; then uses the detection head network of the target detection model to detect the fused multi-scale feature map to obtain a target detection result of the to-be-trained image; finally, determines a loss value based on the target detection result of the to-be-trained image and the annotation information of the to-be-trained image, and uses the loss value to update the parameters of the target detection model to obtain a trained target detection model. The target detection model training method of the embodiment of the present application improves the convertible dilated convolution in the original backbone network to obtain an enhanced convertible dilated convolution, so that the multi-scale feature information in the image can be adaptively extracted, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0059] Figure 1 A flowchart of a target detection model training method according to an embodiment of the present application is shown;
[0060] Figure 2 This is a schematic diagram of the network structure of RestNet50 / ResNet101 in the prior art;
[0061] Figure 3 This is a schematic diagram of the overall process of a target detection model training method in an embodiment of the present application;
[0062] Figure 4 A schematic diagram of a network structure of a SAC in the prior art;
[0063] Figure 5 Schematic diagram of the network structure of an enhanced convertible dilated convolution in an embodiment of the present application;
[0064] Figure 6 A flowchart of a target detection method in an embodiment of the present application is shown;
[0065] Figure 7 This is a schematic diagram of the structure of a target detection model training device in an embodiment of the present application;
[0066] Figure 8 This is a schematic structural diagram of a target detection device according to an embodiment of the present application;
[0067] Figure 9 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0068] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0069] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.
[0070] The present application embodiment provides a method for training a target detection model, such as Figure 1 As shown, a flow chart of a target detection model training method in an embodiment of the present application is provided, and the target detection model training method includes at least the following steps S110 to S150:
[0071] Step S110: Acquire an image to be trained, where the image to be trained includes annotation information of the image to be trained.
[0072] When training the target detection model in the embodiment of the present application, it is necessary to first obtain the image to be trained as a training sample. The target label information Ground Truth is pre-annotated in the training image to serve as the basis for the subsequent calculation of the loss value of the target detection model training. The target detection model training method of the embodiment of the present application can be used for detecting monocular 3D targets. Of course, those skilled in the art can also flexibly expand it to other target detection scenarios, such as 2D detection scenarios, etc., which are not specifically limited here.
[0073] In step S120, the backbone network of the target detection model is used to perform feature extraction on the image to be trained to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained.
[0074] After obtaining the image to be trained, it is necessary to use the backbone network backbone of the target detection model to extract features of the training image to obtain a multi-scale feature map of the image to be trained. The backbone network of the embodiment of the present application can be improved based on the existing RestNet50 or ResNet101 network. The improved backbone network contains enhanced convertible hole convolution, which can adaptively extract multi-scale feature information in the image, that is, realize the extraction of multi-scale features in the shallow network of the target detection model.
[0075] like Figure 2 As shown, a schematic diagram of the network structure of RestNet50 / ResNet101 in the prior art is provided. It can be seen that the original network structure of RestNet50 / ResNet101 contains Conv3x3, that is, 3x3 convolution. When the network structure is transformed, all Conv3x3 can be replaced with the enhanced convertible void convolution of the embodiment of the present application.
[0076] Step S130 , using the feature pyramid network of the target detection model to perform feature fusion on the multi-scale feature map of the image to be trained, to obtain a fused multi-scale feature map.
[0077] Step S140: Detect the fused multi-scale feature map using the detection head network of the target detection model to obtain a target detection result of the image to be trained.
[0078] Step S150: determining a loss value based on the target detection result of the image to be trained and the annotation information of the image to be trained, and using the loss value to update the parameters of the target detection model to obtain a trained target detection model.
[0079] like Figure 3 As shown, a schematic diagram of the overall process of a target detection model training method in an embodiment of the present application is provided. First, the image to be trained and image metadata such as camera intrinsic parameters and image size are input into the improved backbone network backbone, and feature extraction processing is performed in sequence according to C3-C4-C5 to obtain a multi-scale feature map, where each layer in C3-C4-C5 can include the enhanced convertible dilated convolution of the embodiment of the present application, which is used to adaptively extract multi-scale feature information in a shallow network.
[0080] Afterwards, the multi-scale feature maps output by the improved backbone network are input into the Feature Pyramid Networks (FPN) of the target detection model for feature fusion, and the fused feature maps of five levels (P3, P4, P5, P6, and P7) are output.
[0081] Finally, the detection head of the object detection model performs predictions on each fused feature map, outputting a vector containing the target category, target 3D box encoding, target orientation category, target attributes, and target center. The 3D box decoding post-processing module (Box Decoder) decodes the target 3D box encoding vector to obtain decoded information such as target position, target scale, target depth, and target angle. Finally, the final 3D object detection result is determined by combining the score and the NMS algorithm.
[0082] The target detection model training method of the embodiment of the present application improves the convertible dilated convolution in the original backbone network to obtain an enhanced convertible dilated convolution, so that the multi-scale feature information in the image can be adaptively extracted, thereby improving the accuracy of subsequent target detection.
[0083] In one embodiment of the present application, the backbone network also includes a first global context module and a second global context module, the enhanced convertible hole convolution includes multiple stages of convertible hole convolution cascaded in sequence, and the multiple stages of convertible hole convolution include at least the first stage of convertible hole convolution and the second stage of convertible hole convolution, and the backbone network of the target detection model is used to extract features of the image to be trained to obtain a multi-scale feature map of the image to be trained, including: obtaining a first feature map of the image to be trained, the first feature map is output by the upstream module corresponding to the first global context module; using the first global context module to process the first feature map to obtain the second feature map; using the multiple stages of convertible hole convolution cascaded in sequence to process the second feature map to obtain a third feature map; using the second global context module to process the third feature map to obtain a multi-scale feature map of the image to be trained.
[0084] The enhanced switchable atrous convolution (MSAC) in the embodiment of the present application can be regarded as a modification based on the network structure of the original switchable atrous convolution (SAC). Figure 4 This paper provides a schematic diagram of the network structure of a SAC in the prior art. The original SAC network structure only has a single conversion function, which can only convert two dilated convolutions with different dilation rates. Therefore, the receptive field is limited, and the multi-scale feature information that can be extracted is also limited. The SAC network can be specifically represented as:
[0085]
[0086] Among them, x is the input, w is the weight, r is the void rate of the void convolution, which is also the hyperparameter of SAC, Δw represents the weight to be trained, and the conversion function S(〃) is related to the input and position.
[0087] Based on this, the enhanced convertible dilated convolution in the embodiment of the present application may include multiple stages of convertible dilated convolution SAC cascaded in sequence, and the multiple stages are cascaded in sequence, so that it can convert between two or more dilated convolutions with different dilation rates, further expanding the receptive field and enriching the extracted multi-scale feature information.
[0088] like Figure 5 As shown, a schematic diagram of the network structure of an enhanced convertible dilated convolution in an embodiment of the present application is provided. In order to facilitate the understanding of each embodiment of the present application, a cascade of two stages of convertible dilated convolution is used as an example for explanation.
[0089] Specifically, the backbone network of the embodiment of the present application inserts a global context module before and after the enhanced convertible void convolution structure, namely the first global context module (Pre-Global Context) and the second global context module (Post-Global Context). The global context module is similar to SENet (Squeeze-and-Excitation Networks), but has two main differences: 1) The global context module has only one convolution layer and no nonlinear layer; 2) The output is added back to the backbone instead of multiplying the input by the value after the Sigmoid recalibration operation.
[0090] Before the first global context module, the original network structure corresponding to the backbone network before the 3x3 convolution is connected. Its output can be regarded as the first feature map. The first feature map is then processed by the first global context module to obtain the second feature map. The second feature map is then processed by the enhanced convertible hole convolution to obtain the third feature map. Finally, the third feature map is processed by the second global context module to obtain the final multi-scale feature map.
[0091] In one embodiment of the present application, the processing of the first feature map using the first global context module to obtain the second feature map includes: performing global average pooling processing on the first feature map to obtain a first global average pooling processing result; performing 1x1 convolution processing on the first global average pooling processing result to obtain a first 1x1 convolution processing result; and fusing the first feature map and the first 1x1 convolution processing result to obtain the second feature map.
[0092] In one embodiment of the present application, the use of the second global context module to process the third feature map to obtain the multi-scale feature map of the image to be trained includes: performing global average pooling processing on the third feature map to obtain a second global average pooling processing result; performing 1x1 convolution processing on the second global average pooling processing result to obtain a second 1x1 convolution processing result; and fusing the third feature map and the second 1x1 convolution processing result to obtain the multi-scale feature map of the image to be trained.
[0093] The two global context modules of the embodiment of the present application have the same network structure, that is, both contain a global average pooling and a Conv1x1 or 1x1 convolution. Taking the first global context module as an example, the first feature map is first subjected to global average pooling processing, thereby compressing the number of parameters, reducing weights, and reducing the amount of calculation. The first global average pooling processing result is then subjected to 1x1 convolution processing, and finally the first feature map and the first 1x1 convolution processing result are added to obtain the second feature map, which improves information flow and avoids the problem of gradient disappearance and degradation.
[0094] In one embodiment of the present application, the convertible dilated convolution of the first stage includes a first conversion function, a convertible dilated convolution corresponding to a first dilation rate, and a convertible dilated convolution corresponding to a second dilation rate. The convertible dilated convolution of multiple stages cascaded in sequence is used to process the second feature map to obtain a third feature map, including: processing the second feature map using the first conversion function to obtain a processing result of the first conversion function; performing 3x3 dilated convolution processing on the second feature map using the convertible dilated convolution corresponding to the first dilation rate to obtain a first dilated convolution processing result; performing 3x3 dilated convolution processing on the second feature map using the convertible dilated convolution corresponding to the second dilation rate to obtain a second dilated convolution processing result; and fusing the first dilated convolution processing result and the second dilated convolution processing result according to the processing result of the first conversion function to obtain a fourth feature map output by the convertible dilated convolution of the first stage.
[0095] The first stage of the convertible dilated convolution (First Level SAC) in the embodiment of the present application is the same as the aforementioned Figure 4 The structure and parameters of the SAC network provided in the first level SAC are the same. The first level SAC includes the first transfer function S first (〃) and the three main parts of the convertible dilated convolution corresponding to the first dilation rate and the convertible dilated convolution corresponding to the second dilation rate.
[0096] Using the first conversion function S first(〃) The second feature map is processed to obtain the processing result of the first conversion function, which includes S first (x) and 1-S first (x) The two parts, namely the first conversion function, can adaptively adjust and select the dilated convolution of the first dilation rate or the dilated convolution of the second dilation rate for convolution processing according to the learning situation. As an example, the first dilation rate can be set to atrous = 1, and the second dilation rate can be set to atrous = 3. The dilated convolutions all use Conv3x3, i.e., 3x3 convolutions. It should be noted that when atrous = 1, the convertible dilated convolution corresponding to the first dilation rate is essentially an ordinary 3x3 convolution.
[0097] In getting S first (x) and 1-S first (x) After that, the processing result of the first conversion function can be used as a fusion mask, and the S first (x) is multiplied by the first hole convolution result, and 1-S first (x) is feature multiplied with the result of the second dilated convolution processing, and finally the two multiplication results are added to obtain the fourth feature map of the convertible dilated convolution output of the first stage.
[0098] In one embodiment of the present application, the first conversion function includes 5x5 average pooling and 1x1 convolution, and the processing of the second feature map using the first conversion function to obtain the processing result of the first conversion function includes: performing 5x5 average pooling on the second feature map to obtain a first average pooling result; and performing 1x1 convolution on the first average pooling result to obtain the processing result of the first conversion function.
[0099] The first conversion function in this embodiment of the application consists of a 5x5 average pooling layer and a 1x1 convolution layer, which is input- and position-dependent. First, a 5x5 average pooling process is performed on the second feature map to obtain the first average pooling result. This can not only expand the receptive field but also reduce overfitting. The first average pooling result is then subjected to a 1x1 convolution to obtain the processing result of the first conversion function.
[0100] In one embodiment of the present application, the convertible dilated convolution of the second stage includes a convertible dilated convolution corresponding to a second conversion function and a third dilated rate, and the convertible dilated convolution of multiple stages cascaded in sequence is used to process the second feature map to obtain a third feature map, including: processing the second feature map using the second conversion function to obtain a processing result of the second conversion function; performing 3x3 dilated convolution processing on the second feature map using the convertible dilated convolution corresponding to the third dilated rate to obtain a third dilated convolution processing result; obtaining a fourth feature map output by the convertible dilated convolution of the first stage; and fusing the fourth feature map and the third dilated convolution processing result according to the processing result of the second conversion function to obtain the third feature map.
[0101] The second stage of the convertible dilated convolution (Second Level SAC) in the embodiment of the present application includes a second conversion function S nd (〃) and the convertible dilated convolution corresponding to the third dilation rate, in addition, it also includes the output result of the convertible dilated convolution in the first stage, namely the fourth feature map. As an example, the third dilation rate can be set to atrous = 6, and the dilated convolution also uses Conv3x3 to further expand the receptive field.
[0102] Specifically, using the second conversion function S nd (〃) The second feature map is processed to obtain the processing result of the second conversion function, which specifically includes S 2nd (x) and 1-S 2nd (x), which can adaptively adjust and select the dilated convolution of the third dilation rate or the dilated convolution of the first and second dilation rates according to the learning situation. Then the processing result of the second conversion function can be used as a fusion mask, and the S 2nd (x) is multiplied by the fourth feature map output by the convertible hole convolution in the first stage, and 1-S 2nd (x) is feature multiplied with the result of the third dilated convolution processing, and finally the two multiplication results are added to obtain the result of the second stage convertible dilated convolution output, namely the third feature map.
[0103] In one embodiment of the present application, the second conversion function includes 11x11 average pooling and 1x1 convolution, and the processing of the second feature map using the second conversion function to obtain the processing result of the second conversion function includes: performing 11x11 average pooling on the second feature map to obtain a second average pooling result; performing 1x1 convolution on the second average pooling result to obtain the processing result of the second conversion function.
[0104] The first conversion function in this embodiment of the application consists of an 11x11 average pooling layer and a 1x1 convolution layer, which is input- and position-dependent. First, 11x11 average pooling is performed on the second feature map to obtain a second average pooling result. This can expand the receptive field and reduce overfitting. A 1x1 convolution is then performed on the second average pooling result to obtain the processing result of the second conversion function.
[0105] Based on the above embodiments, the MSAC of the embodiment of the present application is a cascade improvement of the original SAC structure. MSAC is designed with two or more switching switches in sequence, which can convert the features converted by SAC and the third void convolution processing result or more void convolution processing results. The two-stage MSAC can be expressed as:
[0106]
[0107] Where x is the input, w is the weight, r_first is the dilation rate of the first stage convertible dilated convolution, which can be set to atrous = 3. Δw_first represents the weight to be trained in the first stage, and the first conversion function S first (〃) consists of a 5x5 average pooling layer and a 1x1 convolution layer. The output of the first stage of the convertible dilated convolution is used as an input of the second stage of the convertible dilated convolution, replacing the original ordinary convolution at that position. r_2nd is the dilated rate of the second stage of the convertible dilated convolution, which can be set to atrous = 6. Δw_2nd represents the weight to be trained in the second stage, and the second conversion function S nd (〃) Consists of an 11x11 average pooling layer and a 1x1 convolutional layer.
[0108] It should be noted that the above-mentioned void ratios atrous = 1, 3, and 6 are all hyperparameters. During actual training, they can be flexibly replaced with other void ratios, such as atrous = 1, 2, and 4, according to training requirements.
[0109] The locking mechanism in the original SAC sets one weight to w and the other weight to w+Δw for the following reason: the target detection model usually uses a pre-trained checkpoint to initialize the weights. However, for the SAC layer converted from the standard convolutional layer, the weights of larger void ratios are usually missing. Since objects of different scales can be roughly detected with different coarseness by the same weight, it is possible to initialize the lost weights with the weights in the pre-trained model. In the embodiment of the present application, w+Δw is used to represent the missing weights starting from the pre-training checkpoint. The w of the convertible convolution at each level is the initial w, Δw_first and Δw_2nd are different, and can be initialized to 0 respectively.
[0110] In one embodiment of the present application, based on Figure 2 The ResNet50 / ResNet101 shown is used as the basic backbone network for feature extraction, in which all Conv3*3 ordinary convolutions can be replaced with the MSAC network structure of the embodiment of the present application. Of course, those skilled in the art can flexibly configure the replacement method according to actual needs. For example, all Conv3*3 are replaced with MSAC, or the first three levels are replaced with SAC convolution, and the three blocks of the last level are replaced with MSAC convolution, etc.
[0111] The present application also provides a target detection method, such as Figure 6 As shown, a flowchart of a target detection method in an embodiment of the present application is provided, and the target detection method at least includes the following steps S610 to S620:
[0112] Step S610, obtaining an image to be detected;
[0113] Step S620, detecting the image to be detected using the target detection model to obtain a target detection result;
[0114] The target detection model is trained based on any of the aforementioned target detection model training methods.
[0115] To verify the detection accuracy of the object detection model trained in this application, the present embodiment used test data to compare the detection results of the object detection model trained in this application with those of the original monocular 3D object detection method. The experiment showed that the mAP (mean Average Precision) of the object detection model trained in this application improved by 1.2 to 1.4 points compared to the original monocular 3D object detection method. Of course, similar methods can also be extended to other backbone networks and similar object detection and segmentation tasks.
[0116] The present application embodiment also provides a target detection model training device 700, such as Figure 7 As shown, a schematic diagram of the structure of a target detection model training device in an embodiment of the present application is provided. The target detection model training device 700 includes: a first acquisition unit 710, a feature extraction unit 720, a feature fusion unit 730, a first detection unit 740 and an update unit 750, wherein:
[0117] A first acquiring unit 710 is configured to acquire an image to be trained, where the image to be trained includes annotation information of the image to be trained;
[0118] a feature extraction unit 720 configured to perform feature extraction on the image to be trained using a backbone network of the object detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained;
[0119] A feature fusion unit 730 is configured to perform feature fusion on the multi-scale feature map of the image to be trained using the feature pyramid network of the object detection model to obtain a fused multi-scale feature map;
[0120] A first detection unit 740 is configured to detect the fused multi-scale feature map using a detection head network of the target detection model to obtain a target detection result of the image to be trained;
[0121] The updating unit 750 is used to determine a loss value based on the target detection result of the image to be trained and the annotation information of the image to be trained, and use the loss value to update the parameters of the target detection model to obtain a trained target detection model.
[0122] In one embodiment of the present application, the backbone network also includes a first global context module and a second global context module, the enhanced convertible hole convolution includes multiple stages of convertible hole convolution cascaded in sequence, and the multiple stages of convertible hole convolution include at least the first stage of convertible hole convolution and the second stage of convertible hole convolution, and the feature extraction unit 720 is specifically used to: obtain a first feature map of the image to be trained, the first feature map is output by the upstream module corresponding to the first global context module; use the first global context module to process the first feature map to obtain the second feature map; use the multiple stages of convertible hole convolution cascaded in sequence to process the second feature map to obtain a third feature map; use the second global context module to process the third feature map to obtain a multi-scale feature map of the image to be trained.
[0123] In one embodiment of the present application, the feature extraction unit 720 is specifically used to: perform global average pooling processing on the first feature map to obtain a first global average pooling processing result; perform 1x1 convolution processing on the first global average pooling processing result to obtain a first 1x1 convolution processing result; and fuse the first feature map and the first 1x1 convolution processing result to obtain the second feature map.
[0124] In one embodiment of the present application, the convertible dilated convolution of the first stage includes a first conversion function, a convertible dilated convolution corresponding to a first dilation rate, and a convertible dilated convolution corresponding to a second dilation rate, and the feature extraction unit 720 is specifically used to: use the first conversion function to process the second feature map to obtain the processing result of the first conversion function; use the convertible dilated convolution corresponding to the first dilation rate to perform 3x3 dilated convolution on the second feature map to obtain the first dilated convolution processing result; use the convertible dilated convolution corresponding to the second dilation rate to perform 3x3 dilated convolution on the second feature map to obtain the second dilated convolution processing result; and according to the processing result of the first conversion function, fuse the first dilated convolution processing result and the second dilated convolution processing result to obtain the fourth feature map output by the convertible dilated convolution of the first stage.
[0125] In one embodiment of the present application, the first conversion function includes 5x5 average pooling and 1x1 convolution, and the feature extraction unit 720 is specifically used to: perform 5x5 average pooling on the second feature map to obtain a first average pooling result; perform 1x1 convolution on the first average pooling result to obtain a processing result of the first conversion function.
[0126] In one embodiment of the present application, the convertible dilated convolution of the second stage includes a convertible dilated convolution corresponding to a second conversion function and a third dilated rate, and the feature extraction unit 720 is specifically used to: process the second feature map using the second conversion function to obtain a processing result of the second conversion function; perform 3x3 dilated convolution processing on the second feature map using the convertible dilated convolution corresponding to the third dilated rate to obtain a third dilated convolution processing result; obtain a fourth feature map output by the convertible dilated convolution of the first stage; and fuse the fourth feature map and the third dilated convolution processing result according to the processing result of the second conversion function to obtain the third feature map.
[0127] In one embodiment of the present application, the second conversion function includes 11x11 average pooling and 1x1 convolution, and the feature extraction unit 720 is specifically used to: perform 11x11 average pooling on the second feature map to obtain a second average pooling result; perform 1x1 convolution on the second average pooling result to obtain a processing result of the second conversion function.
[0128] In one embodiment of the present application, the feature extraction unit 720 is specifically used to: perform global average pooling processing on the third feature map to obtain a second global average pooling processing result; perform 1x1 convolution processing on the second global average pooling processing result to obtain a second 1x1 convolution processing result; and fuse the third feature map and the second 1x1 convolution processing result to obtain a multi-scale feature map of the image to be trained.
[0129] It can be understood that the above-mentioned target detection model training device can implement the various steps of the target detection model training method provided in the aforementioned embodiment. The relevant explanations about the target detection model training method are applicable to the target detection model training device and will not be repeated here.
[0130] The embodiment of the present application also provides a target detection device 800, such as Figure 8 As shown, a schematic structural diagram of a target detection device in an embodiment of the present application is provided. The target detection device 800 includes: a second acquisition unit 810 and a second detection unit 820, wherein:
[0131] The second acquisition unit 810 is used to acquire the image to be detected;
[0132] The second detection unit 820 is configured to detect the image to be detected using a target detection model to obtain a target detection result;
[0133] The target detection model is trained based on the aforementioned target detection model training device.
[0134] It can be understood that the above-mentioned target detection device can implement each step of the target detection method provided in the aforementioned embodiment. The relevant explanations about the target detection method are applicable to the target detection device and will not be repeated here.
[0135] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 9 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.
[0136] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.
[0137] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.
[0138] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a target detection model training device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:
[0139] Acquire an image to be trained, wherein the image to be trained includes annotation information of the image to be trained;
[0140] Performing feature extraction on the image to be trained using a backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained;
[0141] Performing feature fusion on the multi-scale feature map of the image to be trained using the feature pyramid network of the target detection model to obtain a fused multi-scale feature map;
[0142] Detecting the fused multi-scale feature map using the detection head network of the target detection model to obtain a target detection result of the image to be trained;
[0143] A loss value is determined according to the target detection result of the image to be trained and the annotation information of the image to be trained, and the parameters of the target detection model are updated using the loss value to obtain a trained target detection model.
[0144] The above application Figure 1The method performed by the target detection model training device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.
[0145] The electronic device may also perform Figure 1 The target detection model training device is executed in the method, and the target detection model training device is implemented in Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.
[0146] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 The method performed by the target detection model training device in the embodiment shown is specifically used to perform:
[0147] Acquire an image to be trained, wherein the image to be trained includes annotation information of the image to be trained;
[0148] Performing feature extraction on the image to be trained using a backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained;
[0149] Performing feature fusion on the multi-scale feature map of the image to be trained using the feature pyramid network of the target detection model to obtain a fused multi-scale feature map;
[0150] Detecting the fused multi-scale feature map using the detection head network of the target detection model to obtain a target detection result of the image to be trained;
[0151] A loss value is determined according to the target detection result of the image to be trained and the annotation information of the image to be trained, and the parameters of the target detection model are updated using the loss value to obtain a trained target detection model.
[0152] It should be noted that the electronic device of the embodiment of the present application can also be used to perform Figure 6 The method performed by the target detection device disclosed in the illustrated embodiment will not be described in detail.
[0153] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0154] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0157] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0158] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0159] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0160] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0162] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.
Claims
1. A target detection model training method, wherein: The target detection model training method includes: Acquire an image to be trained, wherein the image to be trained includes annotation information of the image to be trained; Performing feature extraction on the image to be trained using a backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained; Using the feature pyramid network of the target detection model to perform feature fusion on the multi-scale feature map of the image to be trained, to obtain a fused multi-scale feature map; Detecting the fused multi-scale feature map using the detection head network of the target detection model to obtain a target detection result of the image to be trained; Determining a loss value based on the target detection result of the image to be trained and the annotation information of the image to be trained, and using the loss value to update the parameters of the target detection model to obtain a trained target detection model; The backbone network further includes a first global context module and a second global context module, the enhanced convertible dilated convolution includes a plurality of stages of convertible dilated convolution cascaded in sequence, and the plurality of stages of convertible dilated convolution include at least a first stage convertible dilated convolution and a second stage convertible dilated convolution, The step of extracting features from the image to be trained using the backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained includes: Obtaining a first feature map of the image to be trained, where the first feature map is output by an upstream module corresponding to the first global context module; Processing the first feature map using the first global context module to obtain a second feature map; Processing the second feature map using a plurality of stages of convertible dilated convolutions cascaded sequentially to obtain a third feature map; The third feature map is processed using the second global context module to obtain a multi-scale feature map of the image to be trained.
2. The target detection model training method according to claim 1, wherein: The using the first global context module to process the first feature map to obtain the second feature map includes: Performing global average pooling on the first feature map to obtain a first global average pooling result; Performing a 1x1 convolution process on the first global average pooling process result to obtain a first 1x1 convolution process result; The first feature map and the first 1x1 convolution processing result are fused to obtain the second feature map.
3. The target detection model training method according to claim 1, wherein: The convertible dilated convolution of the first stage includes a first conversion function, a convertible dilated convolution corresponding to a first dilation rate, and a convertible dilated convolution corresponding to a second dilation rate. The convertible dilated convolution of the second stage is used to process the second feature map by sequentially cascading multiple stages of convertible dilated convolution to obtain a third feature map, including: Processing the second feature map using the first conversion function to obtain a processing result of the first conversion function; Performing a 3x3 dilated convolution on the second feature map using the convertible dilated convolution corresponding to the first dilated rate to obtain a first dilated convolution result; Performing a 3x3 dilated convolution on the second feature map using a convertible dilated convolution corresponding to the second dilated rate to obtain a second dilated convolution result; According to the processing result of the first conversion function, the first void convolution processing result and the second void convolution processing result are fused to obtain the fourth feature map of the convertible void convolution output of the first stage.
4. The target detection model training method according to claim 3, wherein: The first conversion function includes 5x5 average pooling and 1x1 convolution, and the processing of the second feature map using the first conversion function to obtain a processing result of the first conversion function includes: Perform 5x5 average pooling on the second feature map to obtain a first average pooling result; Perform a 1x1 convolution on the first average pooling result to obtain a processing result of the first conversion function.
5. The target detection model training method according to claim 1, wherein: The convertible dilated convolution of the second stage includes a convertible dilated convolution corresponding to a second conversion function and a third dilated rate, and the convertible dilated convolution of the second stage is used to process the second feature map by sequentially cascading multiple stages of convertible dilated convolution to obtain a third feature map, including: Processing the second feature map using the second conversion function to obtain a processing result of the second conversion function; Performing a 3x3 dilated convolution on the second feature map using the convertible dilated convolution corresponding to the third dilated rate to obtain a third dilated convolution result; Obtain a fourth feature map output by the convertible dilated convolution in the first stage; According to the processing result of the second conversion function, the fourth feature map and the third hole convolution processing result are fused to obtain the third feature map.
6. The target detection model training method according to claim 5, wherein: The second conversion function includes 11x11 average pooling and 1x1 convolution, and the processing of the second feature map using the second conversion function to obtain a processing result of the second conversion function includes: Perform 11x11 average pooling on the second feature map to obtain a second average pooling result; Perform a 1x1 convolution on the second average pooling result to obtain a processing result of the second conversion function.
7. A target detection method, wherein: The target detection method comprises: Obtain the image to be detected; Detecting the image to be detected using a target detection model to obtain a target detection result; The target detection model is trained based on the target detection model training method according to any one of claims 1 to 6.
8. A target detection model training device, wherein: The target detection model training device comprises: A first acquisition unit is configured to acquire an image to be trained, wherein the image to be trained includes annotation information of the image to be trained; a feature extraction unit, configured to perform feature extraction on the image to be trained using a backbone network of the target detection model to obtain a multi-scale feature map of the image to be trained, wherein the backbone network includes an enhanced convertible dilated convolution, and the enhanced convertible dilated convolution is used to extract multi-scale feature information from the image to be trained; A feature fusion unit, configured to perform feature fusion on the multi-scale feature map of the image to be trained using a feature pyramid network of the target detection model to obtain a fused multi-scale feature map; A first detection unit is configured to detect the fused multi-scale feature map using a detection head network of the target detection model to obtain a target detection result of the image to be trained; an updating unit, configured to determine a loss value based on the target detection result of the image to be trained and the annotation information of the image to be trained, and to update the parameters of the target detection model using the loss value to obtain a trained target detection model; The backbone network further includes a first global context module and a second global context module, the enhanced convertible dilated convolution includes a plurality of stages of convertible dilated convolution cascaded in sequence, and the plurality of stages of convertible dilated convolution include at least a first stage convertible dilated convolution and a second stage convertible dilated convolution, The feature extraction unit is specifically used for: Obtaining a first feature map of the image to be trained, where the first feature map is output by an upstream module corresponding to the first global context module; Processing the first feature map using the first global context module to obtain a second feature map; Processing the second feature map using a plurality of stages of convertible dilated convolutions cascaded sequentially to obtain a third feature map; The third feature map is processed using the second global context module to obtain a multi-scale feature map of the image to be trained.
9. A target detection device, wherein: The target detection device comprises: A second acquisition unit, configured to acquire an image to be detected; A second detection unit is used to detect the image to be detected using a target detection model to obtain a target detection result; Wherein, the target detection model is trained based on the target detection model training device described in claim 8.